save to one file vs many files

For all coding issues - MODers and programmers, HTML and more.

Moderators: Jeff250, fliptw

Post Reply
User avatar
Isaac
DBB Artist
DBB Artist
Posts: 7127
Joined: Mon Aug 01, 2005 8:47 am
Location: Ơ̸̦͇̲̬̭̱̰͎̞͈̣͎͚̳ͬ͋̃̀̇͊͂͋͐ͦ̽ͣ̂ͥ͊̅̀̚͠ B̶͖̯͉̜̰̲̓̔͋̈́ͅ È̯ Y̪̤̼͉̠̙͝

save to one file vs many files

Post by Isaac » Tue Jan 11, 2011 9:43 am

I have a few lists, each with a key column (like user IDs) and reference columns. I have a few of these that are pulled together using a master list that has one key column and reference column, which the app can quickly pull from.

Is it better to have this all in one file or is it faster for python have separate files, which are read and written during each user's session? Sometimes these sessions run parallel to each other.
User avatar
Jeff250
DBB Master
DBB Master
Posts: 6387
Joined: Sun Sep 05, 1999 2:01 am
Location: ☃☃☃

Post by Jeff250 » Tue Jan 11, 2011 9:08 pm

I don't think I understand the motivating problem, but if you're only reading or changing small bits of the file at a time, then it's generally better to have it spread over multiple files so as to avoid rewriting the rest when you only change a bit.

Having the data spread over multiple files also reduces the chance for file system race conditions, but the problem is still present. Since most Web servers can handle multiple requests--even from the same person--simultaneously, you need to take extra precautions to be robust to these. Suppose I click on 'increment.py' simultaneously in two windows. This could be the order of events:

Process A reads i=10 in counter.txt.
Process B reads i=10 in counter.txt.
Process A writes i=11 to counter.txt.
Process B writes i=11 to counter.txt.

I clicked increment twice, but it only incremented once! In general, you can get around this using file locks, but at this point, instead of reinventing a transactional database, why not use one like Postgres or MySQL. (Postgres seems to be more popular in the Python world in my experience.)
User avatar
Isaac
DBB Artist
DBB Artist
Posts: 7127
Joined: Mon Aug 01, 2005 8:47 am
Location: Ơ̸̦͇̲̬̭̱̰͎̞͈̣͎͚̳ͬ͋̃̀̇͊͂͋͐ͦ̽ͣ̂ͥ͊̅̀̚͠ B̶͖̯͉̜̰̲̓̔͋̈́ͅ È̯ Y̪̤̼͉̠̙͝

Post by Isaac » Sat Jan 15, 2011 3:52 pm

Cool ok. I'll start reading about the two.
User avatar
Xamindar
DBB Admiral
DBB Admiral
Posts: 1497
Joined: Sun Jun 06, 2004 2:44 am
Location: California
Contact:

Post by Xamindar » Sat Jan 15, 2011 4:22 pm

Also, if it is a small need check out sqlite as well. You might not need to have a full fledged database server installed just for what you are doing.
Why doesn't it work?
Post Reply