• All, Gmail is currently rejecting messages from my host. I have a ticket in process, but it may take some time to resolve. Until further notice, do NOT use Gmail for your accounts. You will be unable to receive confirmations and two factor messages to login.

Remove unused music and squeeze size of .pak files

netspider1

New member
Hello,
Many games have bulk size because are stuffed with unused music (.bor .ogg .wav) files inside. Sometimes it reach hundreds of MB  :o
However to walk through every .bor .ogg .wav file and check whether is invoked in the game or not and delete it, is a loooong dirty job. Especially if you have huge collection with 90% of the games stuffed with dead music files inside.  :-\
So, I decided to clean the .pak files from dead music inside that eating gigabytes of space.  ;)
Thereby I made a tools that does it automatically. Might be useful for developers and gamers who wants to shrink the size of .pak files and save a lot of space.  8)
Here you are my 2 tools:

1. clean_music.bat Use this to clean one game at once. First unpack the game (inside in the archive you will find the standard packing/unpacking tools) then copy clean_music.bat in data folder and run it. This will check every music (.bor .ogg .wav) file whether is invoked in every .txt and .c file. If no one is referring the music file therefore that (.bor .ogg .wav) file is orphan and shows deleting file. Also the tool shows every music file in which file is referred and shows a snipped of code linked to the (.bor .ogg .wav) file.
When the tool finish all orphan musical files are deleted, and you can pack the game back. To sum up, this tool require manually to unpack, run the tool and pack the game. Because it is designed when you dealing with only one game. But if you feel to lazy to pack and unpack every game checkout my 2nd tool  ;D

2. clean_music_all_paks.bat Use this tool to clean a bunch of (.pak) games at once. Just copy all the (.pak) game files in tool directory, and run it. That's all, nothing more!
It will clean every single (.pak) game that will find. It will unpack automatically every (.pak) game, will run clean_music.bat to delete the orphan files, and will pack the game back. So when it finish you will have all your (.pak) games cleaned with reduced size. If you have big collection expect to save you gigabytes  8)

If there is any problem, let me know. The tools are not very fast because for every .bor .ogg .wav file is checking every .txt and .c file, so multiply the count of .bor .ogg .wav files by count of .txt .c files and will get the number of iterations, you can do the maths ;)

http://openbor.55freehost.com/Makepak & Extractor.7z
 
It's about time someone came up with this - even just cleaning assets out of your own modules for packing can be a pain. That's probably why few authors bother to do it. :P

Great job! This should be uploaded to the Tools section of Downloads.

DC
 
In this vein I posted a tool somewhere that parses your models.txt and creates a copy of your data/chars/ folder containing only the files that are actually used in your game. Great to release demos that doesn't contain everything you're working on.
 
Good tools netspider!

Just to let you know that some musics are played with script so you might need to read all .c files as well
 
you welcome,
As Bloodbane suggested I added and the .c files in searches.
Also added and .wav files along with .bor and .ogg.
Also exclude a special list of files "Required files" described in the OpenBOR manual here http://www.caskeys.com/arc/games/openbor/wiki/index.php?title=BOR_Manual which must not be deleted no matter is invoked or not
So now it looks for all .bor .ogg .wav files in all .txt and .c files and if nothing found and if not in "Required files" list, will be deleted.
Works like a charm  8) however if you have a game stuffed with a hundred dead .wav files it will take a while to clean it  ;)

It can be easily upgraded and for other resources like .gif .png .jpg (and add an exclusion list of .gif to not delete, as described in the "Required files" section in openBOR's manual)
My primary idea was to clean the orphan music files because they eat the most space, in my 12 GB collection I have many games with 200MB wasted space with dead music files, so now I can strip this junk. Some 400MB-500MB games dwindle to 100MB

It is uploaded on the link above so when the admin want can put my updated tool to download section  :)


@BeasTie I tested  "File in Use Checker" feature in openborstats v067 and the result was:
Final Fight - Heroes v7.4 - lost its entry menu music because music files that should not be removed was removed. openborstats do not care for "Required files" list described in the OpenBOR manual and just delete it. My tool for example do not delete these files and all is OK
Double Dragon 4 - cannot start at all, because many .gif files are removed that should not be removed, they are still in use but openborstats do not recognize keywords like "remap" and so delete these gif files and game cannot start

 
netspider1 said:
It can be easily upgraded and for other resources like .gif .png .jpg (and add an exclusion list of .gif to not delete, as described in the "Required files" section in openBOR's manual)

Those "required" GIF files can now be any image format supported by OpenBOR.  It's common to use PNG for all of them, and a few people might still be using PCX, so you should handle those cases if you're not already.
 
I tested this tool and it works great but you forgot to add data/sounds/go.wav and data/sounds/block.wav as a required file, although mods will still work without them, they are still listed as required files in the manual and i am sure many new and old mods use them.
 
kingherb said:
I tested this tool and it works great but you forgot to add data/sounds/go.wav and data/sounds/block.wav as a required file, although mods will still work without them, they are still listed as required files in the manual and i am sure many new and old mods use them.

Yes, it's an important stuff.

Also, what about a lof file to name all changes in a text file?
Could be a great feature...
 
God deam! You a right!!  Why I cant see it here in documentation? http://www.caskeys.com/arc/games/openbor/wiki/index.php?title=BOR_Manual
if they are used through reference somewhere in .txt or .c files then will not be scrapped. But I checked on double dragon 4 and go.wav is scraped  :'( because no one refer it.
These are important. go.wav when the screen is ready to move next when it says go-> go-> and block.wav is character block move

I will fix the tool. Also I found from the source code in open bor land.wav and plane.bor to be hardcoded as default music files
Why they was not in documentation?!?  Now I have to  fix all my games 11GB from deleted go.wav. Deam  >:(
 
Hmmm.... we might have missed informing those two
Anyways, go.wav is SFX played when arrow (usually pointing right) is shown after wait is cleared
block.wav is the default SFX played when a character blocks an attack
 
I fixed the tool, thank you all guys.
now few more files are protected from deleting: sounds\block.wav  sounds\go.wav    sounds\land.wav  and  music\plane.bor
Also I will re-upload all games without missing these files. I fixed all games and tomorrow will be re-uploaded. From 11GB colectin it cleaned 3GB dead music files  :)
 
In case you missed any others, here's a list of hardcoded filenames in the engine.  I got this by running
Code:
strings OpenBOR.exe | grep 'data/'
. The files without extensions are images or music files where the extension can vary.

data/pal.act
data/sprites/font
data/sprites/fontmask
data/sounds/go.wav
data/sounds/beat1.wav
data/sounds/block.wav
data/sounds/fall.wav
data/sounds/get.wav
data/sounds/money.wav
data/sounds/jump.wav
data/sounds/indirect.wav
data/sounds/punch.wav
data/sounds/1up.wav
data/sounds/timeover.wav
data/sounds/beep.wav
data/sounds/beep2.wav
data/sounds/bike.wav
data/music/remix
data/lifebar.txt
data/menu.txt
data/script.txt
data/scripts/update.c
data/scripts/updated.c
data/scripts/level.c
data/scripts/endlevel.c
data/scripts/keyall.c
data/scripts/timetick.c
data/scripts/loading.c
data/scripts/score1.c
data/scripts/score2.c
data/scripts/score3.c
data/scripts/score4.c
data/scripts/key1.c
data/scripts/key2.c
data/scripts/key3.c
data/scripts/key4.c
data/scripts/join1.c
data/scripts/join2.c
data/scripts/join3.c
data/scripts/join4.c
data/scripts/respawn1.c
data/scripts/respawn2.c
data/scripts/respawn3.c
data/scripts/respawn4.c
data/scripts/die1.c
data/scripts/die2.c
data/scripts/die3.c
data/scripts/die4.c
data/sprites/shadow1
data/sprites/shadow2
data/sprites/shadow3
data/sprites/shadow4
data/sprites/shadow5
data/sprites/shadow6
data/sprites/arrow
data/sprites/arrowl
data/bgs/loading
data/bgs/loading2
data/sprites/hole
data/video.txt
data/videopc.txt
data/music/gameover
data/scenes/gameover.txt
data/bgs/hiscore
data/bgs/complete
data/music/complete
data/bgs/unlockbg
data/bgs/select
data/music/menu
data/bgs/logo
data/scenes/logo.txt
data/scenes/intro.txt
data/scenes/howto.txt
data/bgs/titleb
data/bgs/title
data/bgs/title
data/translation.txt
data/scripts/openbor.h
 
Thanks Plombo. I checked again and I can confirm, all this audio files are excluded from deletion. So the tool will not cripple your game if you use it.  :)
In my 14GB collection I cleaned 3GB dead music  8)

I was wondering to make same tool for pictures. First check all .txt files whether is invoked, and then delete the dead .txt and then check pictures whether are used in some .txt. So when I delete unused .txt, then after the pictures will be deleted too. I am wondering how much space will be free, how many dead pictures are stuffed inside.  ::)

I am playing Open BOR on my phone so few gigabytes matter in phone memory. My phone is Much G2 it has hardware gamepad and buttons on it, but it not looks like game console, it looks like normal 5" phone and no one can suspect that it can be game console too. I play OpenBor in full screen without nasty screen controls above  8)
 
netspider1 said:
I was wondering to make same tool for pictures. First check all .txt files whether is invoked, and then delete the dead .txt and then check pictures whether are used in some .txt. So when I delete unused .txt, then after the pictures will be deleted too. I am wondering how much space will be free, how many dead pictures are stuffed inside.  ::)

This would probably be less advantageous, since pictures, especially the indexed PNGs that OpenBOR uses, are generally much smaller than music/sound files.  Though you could probably save some space by deleting unused animated GIF cutscenes.

netspider1 said:
Thanks Plombo. I checked again and I can confirm, all this audio files are excluded from deletion. So the tool will not cripple your game if you use it.  :)
In my 14GB collection I cleaned 3GB dead music  8)

[...]

I am playing Open BOR on my phone so few gigabytes matter in phone memory. My phone is Much G2 it has hardware gamepad and buttons on it, but it not looks like game console, it looks like normal 5" phone and no one can suspect that it can be game console too. I play OpenBor in full screen without nasty screen controls above  8)

What's this about gigabytes?  Every PAK I've ever seen has been less than 1 GB, and the most memory I've ever seen OpenBOR use was about 200 MB.  And that was for a mod using a ridiculous amount of script
Code:
#include
where it could have been using
Code:
#import
; practically any well-optimized game will take less than 64 MB of memory.

Actually, a great idea for a tool would be something that goes through script files and tells you where you could use
Code:
#import
instead of
Code:
#include
to save memory.  Maybe I should give that a try sometime.
 
Plombo I meant, I have 14GB archived OpenBOR games in total. I run the tool on auto cleaning all games and 3GB was cleaned.  So my collection from 14 GB archive squeeze to 11GB archive.  So 3GB dead stuffed music and audio files was cleaned  :)
The biggest game is "Vermilion Sword - The Legend of Calibur" about 800MB archived, and I shrank to 600MB archived. The next bigger games are about 200-300MB

With Memory on the phone I meant not RAM memory, I meant store memory. I keep OpenBOR games in my phone so few gigabytes less is a matter  ;)
 
I will bump this topic because something got my attention:

data/scripts/die1.c
data/scripts/die2.c
data/scripts/die3.c
data/scripts/die4.c[/quote/

According to the manul, this would be pdie#.c. Do just die#.c works too?

Btw, nice tool.
 
Back
Top Bottom