ProjectsWhat's NewDownloadsCommunitySupportCompany
Forum Index » S.T.A.L.K.E.R.: Shadow of Chernobyl Forum » Mod discussion
Script optimization question

« Previous 10 events | 1 2 3 4 | Next 10 events »| All Messages
Posted by/on
Question/AnswerMake Newest Up Sort by Descending
  20:55:12  29 January 2013
profilee-mailreply Message URLTo the Top
ThunderFreak
Senior Resident
 

 
On forum: 08/07/2009
Messages: 685
You got mail.
  21:06:28  29 January 2013
profilee-mailreply Message URLTo the Top
Decane
Senior Resident
 

 
On forum: 04/04/2007
Messages: 1701
Wow, thanks a million! I read your comments and they are way more detailed than I could have expected; that was a huge help. Very kind of you.
  00:22:24  30 January 2013
profilee-mailreply Message URLTo the Top
Alundaio
Sad Clown
(Resident)

 

 
On forum: 04/05/2010
 

Message edited by:
Alundaio
01/30/2013 4:48:47
Messages: 2230

---QUOTATION---
One hint to boost your code a little bit more.

NEVER declare a variable inside a for ... end block.

---END QUOTATION---



Yeah that is important. Too bad GSC didn't practice this. In CoP, gulag_general.script is RIDDLED with these.


function is_bandit_happy()
	local bandit_mood = sim_faction.get_mood("bandit")
	return bandit_mood ~= "depressed" or  bandit_mood ~= "a bit blue" or bandit_mood ~= "content"
end

  03:45:50  9 February 2013
profilee-mailreply Message URLTo the Top
NatVac
Senior Resident
 

 
On forum: 06/15/2007
Messages: 4282
>> NEVER declare a variable inside a for ... end block.

Um... actually, it really doesn't matter.

First some background on what follows: A long while back Decane (!) posted a set of links for learning Lua. Here's some info from the Lua users wiki he linked.

Locals are faster than globals, as Thunderfreak first said:
http://lua-users.org/wiki/OptimisingUsingLocalVariables

This is true even if there are register spills (when there are more vars than CPU registers) resulting in memory storage.

Local functions are slower when encountered in execution (i.e., don't declare functions inside loops)
http://lua-users.org/wiki/MinimisingClosures

Next, some contemplations that might impact what really happens with locals in loops:

Lua is designed for gaming. That implies speed of execution. To this end, most Lua deployments including STALKER use the just-in-time compiler. This means that each line is not continually being interpreted.

Most stack-based compilers (all?) simply pre-allocate on the stack all locals declared inside a function. This means that with the logical exception of closures (those aforementioned functions that have additional info associated with their context, making them dynamic), all local variables are simply indexed off the stack and it doesn't matter where they are declared inside the function.

Note that the stack in modern compilers is smart, using CPU registers for some stack items. Quantity of local vars might matter (see "register spills" above), but their impact should still be constant per n local variables.

Finally, some empirical observations:

I created a test with two local variables of interest, one declared outside the outer of two nested loops and one declared inside the inner loop. The loops were for ix=i,100,1 do for id_=1,4000000,1 do [...] end end, with the [...] containing enough repeated assignments of simple-addition calculations to the locals (10 statements) to make the process last several seconds.

There was no discernible difference. Each execution took just under seven seconds. (Slow box.)

I then added another local variable to each loop, so that the first test had two outside locals and the second had two inside locals declared and assigned. The statements were changed to share the variables but no new statements were added.

No change in test results. Each execution still took just under seven seconds.

In the second test two locals were declared in the inner loop and assigned a initial calculated result 400 million times. (With the ten statements, over four billion statements were executed.)

In a third test, I moved the local declarations of the first test outside the function. It took almost a minute to execute the same code, further supporting Thunderfreak's first reply.

No, that's not rigorous science, but it is more in the ballpark than any speculation.

It may be that some of your tests may experience an initial overhead of parsing/compiling. I expected a need to pre-load the test script, but the game was already doing that for me; my test framework lists the available utilities and their versions, which requires that the scripts be valid. So if you do this test on your own, run it several times.

Conclusion: Don't use local functions inside functions. Other local variables are fine, with the exception of table creation.

With that said, this is mainly a "peephole" optimization discussion. If we are talking about local tables, there's a trade-off. While it doesn't matter where local tables are declared within a function, it may be sometimes better to hoist constant-data tables out of the function and set local vars equal to the external tables. This results in a small hit when loading, but removes the hit incurred on each call to the function using the table.

There are a lot of good thought-stimulating ideas here. I would like to add a couple of my own: I believe there are many important areas ripe for optimization that might have a greater impact. One is large-resource management. The moment virtual-memory usage demands swapping external (non-RAM) storage is the moment the framerate plumments in-game. The brain-dead prefetch was loading in everything, resulting long level loads followed by lag anyway due to dumping unused stuff to load in needed stuff. Not to mention that some vanilla prefetch resources are not even used by the game...

The game is rife with stuff not used. It loads logic for test objects and puts the checks for those cases in the smart terrain assignments. It checks for specific entities as part of tests that were not removed. There are lots of extra support for demos and dev-specific explorations that the game loads and then processes each time, like info_portions and XML structures. Some is simply loaded in memory, some is constantly being checked.

>> "friend" and "enemy" are the only possible values for the variable db.actor

If we are talking SoC post init I think you mean this as a shorthand for db.actor.something or db.actor:something() as db.actor holds a game object.

If it's either/or, I recommend using a boolean, not a string compare (slower):

function is_friend() return db.actor:relation(npc) == game_object.friend end

If doing if-then-else using the same db.actor:relation(npc), assign it to a local var first and use that in the comparisons. This is only slightly slower if the first "if" test is true, and much faster for subsequent tests.

>> There is a myth that ipairs is faster then pairs

I've never heard this myth. I've only heard that ipairs is for accessing the numerically-indexed members of a table in sequential order. There is no guarantee on order for pairs. Also, tables can have non-numerically-indexed items combined with numbered ones; I think ipairs will ignore the ones not numbered.

Due to your comments here, guys, I'm already looking at more optimizations myself. Similar to Alundaio's quote from Donald Knuth, there's M.A. Jackson's Rules of Optimization:

Rule 1: Don't do it.
Rule 2 (for experts only): Don't do it yet.
  21:30:29  10 February 2013
profilee-mailreply Message URLTo the Top
Decane
Senior Resident
 

 
On forum: 04/04/2007
Messages: 1701
NatVac, thanks for your comments. On their account, I now have another question: how do you go about identifying which files are not used by the game at all? I would like to do something similar to your 'SHOC Treatment' project for Clear Sky, but at present I am pretty much limited to commenting out redundant debug code like printf() and print_table() since I don't know of any way to batch-identify unused functions.

Actually, it would be convenient to have a method of identifying unused sounds, textures and meshes in addition to unused scripts; particularly since the CS database files are filled with unused SOC assets.
  11:52:36  18 February 2013
profilee-mailreply Message URLTo the Top
NatVac
Senior Resident
 

 
On forum: 06/15/2007
Messages: 4282
My comments about the "local vars in loops" test might have been a bit confusing. The point was that the location of local var declarations inside a function didn't matter (inside nested loops or outside), but vars declared outside the function are much more slowly accessed.

>> how do you go about identifying which files are not used by the game at all?

It's mostly informed trial-and-error. Here are some of my techniques for SoC which might apply in some fashion to the other games. Some are probably mentioned in this thread and elsewhere already. Of course, one should make backups and test each change or group of changes before going on to the next. Keep a list of changes; some change-related problems might become apparent only after extended play or later in the game.

Identifying the unused resources:

I use Notepad++ to search in all resource files. I locate the starting points for the processing of data; in SoC I traced all the loading of nested files via #include directives. I have an extracted base of resources in a single directory for the different patch versions, but the primary search is in the 1.0004 gamedata subdirectory as the later versions don't add any significant stuff. Thisincludes an extracted spawns subdirectory.

You need a smart extraction from the database files, as the files must be complete up to that version and the newer must supplant the older. I remember that items.ltx and at least one other config or script file were not in gamedata.dbb the way you would expect.

I use search filters to limit which files are checked.

I use the "Execute Lua Script Command" utility in the ZRP Support Utility toolkit. You can type in the name of the script file and press Enter to see if the script is valid. It shows up as a table if it is valid, otherwise it shows "nil". Such scripts are not used by the game. (The game might try to use it, but that will be a crash.) So a quick way to identify broken files would be a script loop that would check a script file for existence (example: xxxx.script) and then check to see if there is also a table for it (type(xxxx) == "table" or at least not nil).

You can't know that a script is loaded by the game that way because of an effect similar to the Heisenberg uncertainty principle: The very act of checking a script object in Lua will load that file if it is not already loaded.

Special note: When determining if a resource file is used, removing files from a gamedata directory isn't enough. You have to remove them from the database files as well. A possible workaround is to use a zero-length file in gamedata\.

First I look for test code. Extremely wasteful tests occurred in code for a particular entity (e.g., esc_wolf) that would only print out something. I look at the files. Some are suspicious just by their names, like the x1.script through x18.script files, or the task_number.script files.

You can embed debug statements in script files to see if/when they are called. Statements at file scope (i.e., in the file but not in any function) are executed when the file is parsed successfully.

You can delete the suspicious script and run the game. It will complain if it can't find the script. See the special note above.

Other script files can be checked by searching all script files for their name sans ".script".

I was able to remove about half the scripts from SoC as unused or unneeded. Similar tactics can be used for textures and meshes, although some of the references are in the non-text files. The game can help by crashing with a "can't find" message.

I would expect CS to not have most of the unused SoC junk while it adds new and unused junk of its own.

After the unused files are removed:

It helps greatly to learn the system. Knowing that certain resources are not used can help identify what can be further pruned, like gulag references to Dead City or kishka in SoC's config\misc\gulag_tasks.ltx, or test logic in the same file.

At some point you will start asking yourself, "Do NPCs really need a xr_test scheme?" and start experimenting with reducing the load. I found out that almost all NPCs were being assigned sounds only used by special stalkers. Each NPC is now only burdened with about 40% of his original load in the recent ZRP versions.

>> at present I am pretty much limited to commenting out
>> redundant debug code like printf() and print_table()

Those can actually work in CS, but that change will probably go a long way to reclaiming time and memory.

Related stuff for commenting out/removing:

Statistics and collection of any data that is not used except by the printing/logging functions. I get a kick out of a modder's "Eureka!" when he discovers that the game_stats tables built in bind_stalker.script are the source of "leaks" -- they are not, but because they just keep accumulating stats, your memory usage grows and grows in each game session. And the smart terrain stats were always loaded but only used by folks who enabled the display of "spots".

Code that encloses only printing/logging statements. Often a function will declare a local variable, assign it a complex value, then only use it in a subsequent print statement, or there will be a block that tests a value only to print it. You can safely remove all of that.

You can also try to identify bottlenecks and memory wasters. Use debug statements liberally. As discussed previously in this thread, sometimes a simple change to the way that code executes can speed up execution. I don't know about the other games, but prefetch in SoC was not very useful, making for a long load only to result in unused data getting swapped out anyway.

>> particularly since the CS database files are filled with unused SOC assets.

So are the SoC database files.

And xoen, bamah and others showed the value of compressing many of the textures for a faster load and less memory footprint without a noticeable loss of visual quality.
  05:21:57  10 March 2013
profilee-mailreply Message URLTo the Top
Alundaio
Sad Clown
(Resident)

 

 
On forum: 04/05/2010
Messages: 2230
Lua optimization: http://stackoverflow.com/questions/154672/what-can-i-do-to-increase-the-performance-of-a-lua-program
  23:42:07  8 April 2013
profilee-mailreply Message URLTo the Top
NatVac
Senior Resident
 

 
On forum: 06/15/2007
Messages: 4282
Thanks for linking that, Alundaio! I've lost a lot of time since I saw this almost a month ago, converting string concats in SoC in quite a few files and I'm still not done.

SoC's NPC logic is largely dynamically assigned and a large part is constructed on-demand with string concatenations in loops. My testing on the changed code shows some memory savings but not necessarily any processing savings. I'm far from done with the changes and testing, though.

The rest of the linked optimizations I'd seen before, but the argument for table.concat() was quite persuasive.
  00:42:02  9 April 2013
profilee-mailreply Message URLTo the Top
LoNer1
Настоящий Cталкер, Русская Версия
(Resident)

 

 
On forum: 10/23/2009
 

Message edited by:
LoNer1
04/09/2013 0:42:51
Messages: 1890

---QUOTATION---
Thanks for linking that, Alundaio! I've lost a lot of time since I saw this almost a month ago, converting string concats in SoC in quite a few files and I'm still not done.

SoC's NPC logic is largely dynamically assigned and a large part is constructed on-demand with string concatenations in loops. My testing on the changed code shows some memory savings but not necessarily any processing savings. I'm far from done with the changes and testing, though.

The rest of the linked optimizations I'd seen before, but the argument for table.concat() was quite persuasive.
---END QUOTATION---



NatVac, I have a slight mancrush on you, seriously

You keep this community alive, constantly replying and updating your flawless additions I don't have any input here, but thanks! I just know there's good stuff going on when the last person to reply to 6 threads is you!
  05:03:00  27 April 2013
profilee-mailreply Message URLTo the Top
NatVac
Senior Resident
 

 
On forum: 06/15/2007
Messages: 4282
There's been some reporting that shaders can affect stuttering. Supposedly getting improved shaders can reduce the pauses considerably. I can't tell yet.

¯¯¯¯¯¯¯¯¯¯
Thanks, I think, LoNer1. But the community is surviving just fine without my being here that often. There are a lot of consecutive posts by me because I don't post that often, and mainly catch up on the unanswered ones when I get a large block of time to reply.

And I'm still human, last I checked. My errors are compounded when I've missed a lot of sleep, so it is good that you guys can catch what I miss.
 
Each word should be at least 3 characters long.
Search:    
Search conditions:    - spaces as AND    - spaces as OR   
 
Forum Index » S.T.A.L.K.E.R.: Shadow of Chernobyl Forum » Mod discussion
 

All short dates are in Month-Day-Year format.


 

Copyright © 1995-2020 GSC Game World. All rights reserved.
This site is best viewed in Internet Explorer 4.xx and up and Javascript enabled. Webmaster.
Opera Software products are not supported.
If any problem concerning the site functioning under Opera Software appears apply
to Opera Software technical support service.