File Organization

edited January 7 in General
As you probably know, AutoTRAX is a self-contained file format, in that all data is contained in a single file and there are no external dependencies on other files. This ensures integrity and avoids any devastating errors where you have missing files and therefore cannot reconstitute your project, resulting in a complete disaster. This feature also allows you to pass a design onto others as a single self-contained file.

In addition, AutoTRAX's  history features where absolutely every change you do while creating your PCB are part are recorded in a special directory with the same name as the project and an extension of ‘.history’. One of the great features of this history directory is that undos can be done even after you have exited a design session. Undos can go back to changes that occurred in previous sessions. It also has the feature that if you failed to save your design, when you restart AutoTRAX you can use redo to reinstate all the changes that you made to your design before you failed to save it, how cool is that? 

I have received requests to store some parts of the design in external files such as imported 3-D models. I have looked into this and done some experiments and I found that it could reduce the size of each history file by 0-90%. But the problem with this is that the project file is no longer self-contained and produces a great risk of you losing your 3-D models. Also, the current practice of storing the 3-D models in each of the history files makes each history file self-contained and capable of being used as a stand-alone project file. This redundancy, I believe, is well worth the extra space it consumes on your disk drive. It makes your design experience more comfortable in certain with a greatly reduced risk of losing your work.

My experiments involved using a compressed zip file as a virtual file system that can hold not only the project file but also the 3-D model file and the complete set of history files. But this makes your work even more susceptible to file corruption. If the compressed zip file became corrupt, then that’s all your work gone. So, using a compressed zip file is not a feasible alternative.

The developer of AutoTRAX DEX. 
·

Comments

  • disc space is cheap but unfortunately SSD space is less so.  Suggested options for people with a real issue :
    1) option to store the history on a different drive
    2) option to limit the size of the stored history

    p.s. I think self-contained files and complete history are great
    ·
  • We are not anymore in 1983 where a PC with 20MB was a luxury
    In 2020 who cares about hard drive size?
    Let's allow Iliya to focus in the real deal
    ·
  • I think the redundancy is a very nice feature, the additional disk space is no problem, today even SSD’s are relatively cheap (if you don’t go for the extremes).

    I remember my first 1GB HDD in mid-90’s that costed more than my 512GB Samsung 960 EVO.

    ·
  • edited January 7
    Let’s do the math?

    So say we are developing a project that will end up being 2 MB in size.

    Also let’s assume that the size of the project file increases linearly over time and so the average file size will be 1 MB.

    Now let’s say that during the project development there were 1000 changes.

    This would make the disc space used to be 1 GB.

    Now, this project is a very significant part of your work and so you would probably wish to allocate sufficient disc space.

    My current surface pro which is an M3 and only as 64 GB of SSD. I would happily let AutoTRAX take-up 1 GB of the disc space.

    You can also some of the history files to free up space if you need. You can even copy them to a separate drive and if you wish to cover them back at a later date.

    You can also, if you like to sail close to the wind, disable the history function, though I would strongly advise against it.

    What would you say if you disable the history function and for some reason your current project was not to your liking, such as a change that you made and not a file corruption. Then, suddenly there is no going back.
    The developer of AutoTRAX DEX. 
    ·
  • Hi Iliya, All,

    Could you provide an option to archive all history except for the last {variable number} say 20..

    What I mean is NOT automatic (although if you could write that without impacting speed and integrity it would be OK..) is a button for ARCHIVE HISTORY and either have a setting for the number of non archived records or enter that number when you select archive.

    Let’s say the number of non archived records is 20..
    I select ARCHIVE HISTORY and all history gets compressed into a single ZIP except the last 20 records and then deleted from the history directory.

    This way you will have a ZIP file with 20 other individual history files..

    The history files will build up as more work is done and when you feel like you want to re archive press the ARCHIVE HISTORY button again and add any new history to the ZIP and delete them.

    I tend to do that manually but I end up with 1000’s of files so I end up deleting the history when I am at a stage I am happy with.

    Regards,

    Mick 

    ·
  • edited January 8
    All the history files are already compressed.

    A better strategy might be to save the current project to a project with the same name but with a postfix number. E.g. same design1 as design2.

    So a separate history directory will be created for design2.

    Doing this keeps a complete history of all your designs. Also each design stage will have a reduced history count.

    I also suggest you back these up to ideally a CD or DVD or even a USB stick, although I would strongly advise using a read-only device such as a DVD, this will protect you from a ransomware attack.

    I strongly advise you to never to delete your history files. You may come to regret it one day.

    How much does a DVD cost, and how much do you value your time?

    Backups will save your life one day.

    The developer of AutoTRAX DEX. 
    ·
  • Ransonware

    Backup to read-only media and media not always connected to your computers. DVD!

    See the woes of Travelex.

    OneDrive won't save you.





    The developer of AutoTRAX DEX. 
    ·
  • I guess it is true that you don´t value backup before you lose valuable data.

    ·
  • Great that you look into this Iliya. I agree on most arguments they just not show the whole picture.
    1. Example calculation is done on just one project. I soon have 20 projects there I like to save history to maybe 10.
    2. Calculations might be wrong. I am fairly certain my history files becomes larger than that.
    3. You forget all parts. Each of them have their own history file.

    All this together means that a project with 5 new custom parts can be big. With today's hard drive sizes and prices this light be a minor problem but do you guys never take backup? At home I have an automatic backup to a NAS and at work I always work on a server. In both situations 5-10 Gig is to much to deal with because of the backup system.

    I can't say what solution that can be best but size of history files has been discussed before
    ·
Sign In or Register to comment.