Xml or Sqlite, When to drop Xml for a Database?

I really like Xml for saving data, but when does sqlite/database become the better option? eg, when the xml has more than _x_ items or is greater than _y_ MB?

I am coding an rss reader and I believe I made the wrong choice in using xml over a sqlite database to store a cache of _all_ the feeds items. There are some feeds which have an xml file of ~1mb after a month, another has over 700 items, while most only have ~30 items and are ~50kb in size after a _several_ months.

I currently have no plans to implement a cap because I like to be able to search through everything.

So, my questions are:

1. When is the overhead of sqlite/databases justified over using xml?
2. Are the **few large xml files** justification enough for the database when there are **a lot of small** ones, though even the small ones will grow over time? (a long _long_ time)

**updated** (more info)

Every time a feed is selected in the GUI I reload all the items from that feeds xml file.

I also need to modify the read/unread status which seems really hacky when I loop through all nodes in the xml to find the item and then set it to read/unread.

To me it really depends on what you are doing with them, how many users/processes need access to them at the same time etc.

I work with large XML files all the time, but they are single process, import style items, that multi-user, or performance are not really needs.

SO really it is a balance.

I say it's not a matter of data size, but of data type. If your data is *structured*, use a relational database. If your data is *semi-structured*, use XML or - if the data amounts really grow too large - an XML database.

I wouldn't use XML for storing RSS items. A feed reader makes constant updates as it receives data.

With XML, you need to load the data from file first, parse it, then store it for easy search/retrieval/update. Sounds like a database...

Also, what happens if your application crashes? if you use XML, what state is the data in the XML file versus the data in memory. At least with SQLite you get atomicity, so you are assured that your application will start with the same state as when the last database write was made.

XML is best used as an interchange format when you need to move data from your application to somewhere else or share information between applications. A database should be the preferred method of storage for almost any size application.

If your searching go with a db. You could split the xml files up into directories to ease seeking, but the managerial overhead easily gets quite heavy. You also get a lot more than just performance with a sql db...

I agree with @Bradley.

XML is very slow and not particularly useful as a storage format. Why bother? Will you be editing the data by hand using a text editor? If so, XML *still* isn't a very convenient format compared to something like YAML. With something like SQlite, queries are easier to write, and there's a well defined API for getting your data in and out.

XML is fine if you need to send data around between programs. But in the name of efficiency, you should probably produce the XML at sending time, and parse it into "real data" at receive time.

All the above means that your question about "when the overhead of a database is justified" is kind of moot. XML has a way higher overhead, all the time, than SQlite does. (Full-on databases like MSSQL are heavier, especially in administrative overhead, but that's a totally different question.)

1. Use XML for data that the
application should know -
configuration, logging and what not.
2. Use databases(oracle, SQL server etc) for data that the user
interacts with directly or
indirectly - real data
3. Use SQLite if the user data is more
of a serialized collection - like
huge list of files and their content
or collection of email items etc.
SQLite is good at that.

Depends on the kind and the size of the data.

If any time you will need to scale, use databases.

Don't forget that you have a great database at your fingertips: the filesystem!

Lots of programmers forget that a decent directory-file structure is/has:

1. It's fast as hell
2. It's portable
3. It has a tiny runtime footprint

People are talking about splitting up XML files into multiple XML files... I would consider splitting your XML into multiple directories and multiple plaintext files.

Give it a go. It's refreshingly fast.