ProjectsWhat's NewDownloadsCommunitySupportCompany
Forum Index » S.T.A.L.K.E.R.: Clear Sky Forum » Mod discussion
Modified LuaJIT library

1 2 3 4 5 6 | Next 10 events »| All Messages
Posted by/on
Question/AnswerMake Newest Up Sort by Descending
  20:15:35  3 October 2013
profilee-mailreply Message URLTo the Top
Xetrill
(Senior)
 
On forum: 07/08/2008
 

Message edited by:
Xetrill
10/04/2013 11:07:43
Messages: 129
Modified LuaJIT library

Hi, I though since I haven't posted in a while I'll let the few interested people know that I'm still working on an improved LuaJIT library.
I have learned quite a bit about LuaJIT's internals since my last post and made some progress.

Here's an overview of what I've done so far:
math:
- Made sure SSE2 is used on Intel and not-used on AMD processors (only relates to Microsoft's C Runtime; msvcrt). Because on AMD x87 FPU instructions are actually faster than SSE2.
- New SSE2 and x87 functions: clamp, round, min, max, rad, deg, sqrt, log2, log10.
- New SSE4.1 and x87 functions: round, floor, ceil.
debug:
- _G.debug is now always registered.
io:
- New functions: cwd (returns current working directory = where fsgame.ltx resides), exists
package:
- Removed never used loaders.
string:
- New functions: trim, ltrim, rtrim, join, joinv, joinh, split. These could all easily be implemented in pure Lua, but the C implementation comes with a significant performance boost. Because of the use of a mutable string buffer.
table:
- New functions: len (reliable version of getn), create (allocates a table, see Lua C API lua_createtable).
- New functions to work with hashmaps/tables: count, min, max, sum, find, any, all, map, retain.
- New functions to work with vectors/lists: countv, minv, maxv, sumv, findv, anyv, allv, mapv, retainv.
marshal: (https://github.com/richardhundt/lua-marshal)
- New library _G.marshal: This is a serialization library and can be used to save and load data without any arbitrary and completely ridiculous limitations as in the games save-system... (similar to xs_save/xs_storage but in C ).
- New functions: encode, decode, clone.
buffer: (modified version of https://github.com/starwing/lbuffer)
- New library _G.buffer: This implements an mutable string buffer in C. This enables fast and efficient string manipulations (one of Lua's weaknesses).
- New functions: append, assign, byte, cap, char, clear, cmp, copy, dump, equals, find, format, free, gmatch, gsub, insert, ipairs, isbuffer, len, lower, ltrim, match, move, new, quote, remove, rep, reverse, rtrim, split, swap, tohex, topointer, tostring, trim, trim_excess, upper.

I intend (already have actually) to add Shiny too, but as it doesn't record anything. So this needs investigation, it's most likely some LuaJIT internal problem, like the hooking not being allowed.
(Shiny: http://code.google.com/p/shinyprofiler/)

Well, that's it for now. If you have any suggestions/wishes/etc., let me know.

Edit: 1 word.
  09:25:01  4 October 2013
profilee-mailreply Message URLTo the Top
SetaKat
Ex modder, Zones only ferret and will someday release a game
(Resident)

 

 
On forum: 02/20/2010
Messages: 6340
1 word.

Awesome!

Can ditch some of my lua coded util functions when you release this.
  19:06:58  12 October 2013
profilee-mailreply Message URLTo the Top
Xetrill
(Senior)
 
On forum: 07/08/2008
 

Message edited by:
Xetrill
10/12/2013 19:08:40
Messages: 129
Small update:
Good News: Shiny is working now. I had to make changes to Shiny and ShinyLua to make it work as well as be concise.

Bad News:
To get Shiny to work with LuaJIT requires that jitting be disabled, that means only the interpreter can be profiled. There is no workaround here, since LuaJIT doesn't call Lua hooks from traces (jitted code paths); for good reason.
So, I just disable jitting when Shiny.start() is called an re-enable it when Shiny.stop() is called.

Sample output from Shiny: http://pastebin.com/XaecXw5r

For any modder interested, here's the current and experimental release:
SkyDrive: http://sdrv.ms/16DGduV (LuaJIT_EXPERIMENTAL.rar)
Dropbox: https://db.tt/6RF6CZRw (maybe it works this time?)

Experimental means, that I haven't written a test-suite yet and thus cannot make any promises.

It contains x87 (AMD) and SSE(2, 3, 4.1; Intel) versions of LuaJIT and ShinyLua (just called Shiny).
A test script that can list all available functions.
And a batch script to start CS and capture stdout output -- this needs to be modified for your setup.
  23:58:28  17 October 2013
profilee-mailreply Message URLTo the Top
SetaKat
Ex modder, Zones only ferret and will someday release a game
(Resident)

 

 
On forum: 02/20/2010
Messages: 6340
Can you provide some sample code for the buffer and marshall classes/functions please?
  12:21:34  18 October 2013
profilee-mailreply Message URLTo the Top
Xetrill
(Senior)
 
On forum: 07/08/2008
 

Message edited by:
Xetrill
10/18/2013 14:53:32
Messages: 129
Ya, documentation/samples always come last .

marshal
For marshal at least, I can refer you to the author because I didn't make any changes -- https://github.com/richardhundt/lua-marshal.

I plan to use just like my xs_storage.script/xs_save.script now.

The general idea is very simple, you call marshal.encode pass it a table you want to save, and it gives you back a blob of data that you can write to disk.

I may chance it to directly write to disk, say given an optional second/third FILE* argument (io.open).

buffer
While I changed buffer, that's mostly internal. Other than removing some functions (pack/unpack and related). It doesn't have much documentation though...
Well, you can look at https://github.com/starwing/lbuffer/blob/master/test.lua to see how it's used.

Mostly it works just like that, expect of course for functions that I have added ([l|r]trim, split, trim_excess, cap) or removed.
Most notably, I removed the library's metatable which means you have to use _G.buffer.new() instead _G.buffer() to create a new buffer.

The internal changes, are to reduce the amount of reallocating. Which as it turns out this library does a lot.
I have no idea what the original author had in mind, when he made it so that buffers can shrink and grow an every single call. Which defeats the whole purpose of a such a buffer...
So I changed that, buffers now only grow and never shrink unless explicitly wished. That can be done by either calling cap(new_capacity) (formerly alloc()) or trim_excess() which will shrink a buffer to it's current used length.

Legend
xs_storage.script: http://pastebin.com/R8bCc7n1
xs_save.script: http://pastebin.com/P7RbJd3y
Other related scripts are in my pastebin: http://pastebin.com/u/Xetrill

Edit: I forgot, there's no need to do "lib = require(lib)" because these libraries get registered implicitly.
  20:15:03  29 October 2013
profilee-mailreply Message URLTo the Top
Xetrill
(Senior)
 
On forum: 07/08/2008
 

Message edited by:
Xetrill
10/31/2013 16:55:14
Messages: 129
The latest version will now always be available through: http://bit.ly/1bG6SXl (MEGA).
  18:47:46  19 December 2013
profilee-mailreply Message URLTo the Top
Jketiynu
Swartz
(Resident)

 

 
On forum: 04/05/2007
Messages: 867
*Bump*

First of all, awesome work.

I'm just wondering though, why don't any of the experimental releases work with COP? I'm sure they work with CS but with COP all of them cause the game to crash rather quickly.
  21:44:55  19 December 2013
profilee-mailreply Message URLTo the Top
Xetrill
(Senior)
 
On forum: 07/08/2008
Messages: 129
Well, because I hadn't bothered actually testing it. Instead guessed based on inspecting DLL im- and exports.
Just another case of an bad assumption; I do make those...

I didn't have much time with the project lately and even then been busy with writing my own Lua C libraries. The lbuffer library as it turns out, has some weird-ass assumptions, that truly baffle me.
Like, what's the point of a mutable string lib that re-allocates on almost every operation?! Apart form that, it was needlessly complicated for what it accomplishes. And couldn't be used from C without a lot of rewriting. So, it's gone and instead I wrote a new one.

Anyway, making it work for CoP isn't as important right now as making it work in general and be useful.
  07:21:53  22 December 2013
profilee-mailreply Message URLTo the Top
Alundaio
Sad Clown
(Resident)

 

 
On forum: 04/05/2010
Messages: 2230
The io and string functions would be extremely helpful. One of Stalker's biggest slowdown is all the string manipulation going on. That alone is worth all your effort. Bravo.
  22:01:19  25 December 2013
profilee-mailreply Message URLTo the Top
Xetrill
(Senior)
 
On forum: 07/08/2008
 

Message edited by:
Xetrill
12/25/2013 22:05:34
Messages: 129
Uploaded a new version today; still experimental though. The new version contains debug (contains debug symbols) and a release build, as well as some scripts and updated syntax highlighting for Sublime Text.

buffer
As mentioned earlier, the lbuffer library is gone and won't ever come back.
The new library (named lxs_string and lxs_buffer), allows for more and precise (if wished) control over memory allocation. And generally assumes the user knows what he/she is doing.
It's still easy and straightforward to use. But it's not years old and doesn't yet have a complete test suite -- I know that buffer.cap() is currently bugged.

The main thing is, buffers don't shrink, ever. Or explicitly told so, this means buffers, only ever grow. But only up to a certain point, currently two limits are in place. One for checking 'smaller' increments, to try to catch errors early.
And one for the total accumulation of memory used.
Of course, if a buffer turns black it's memory will get reclaimed.

The API at the moment, in alphabetical order:
append, cap, clear, equals, insert, isbuffer, len, lower, ltrim, new, remove, rep, replace, reverse, rtrim, split, substr, tostring, trim, trim_excess, upper, (__concat, __eq, __gc, __len, __tostring).

Many functions accept ranges (offset and count/length) in a Lua fashion, meaning 1-based indexes.

string
Moved join* functions to _G.table.

The additional APIs at the moment, in alphabetical order: ltrim, rtrim, split, trim.

table
Renamed a lot here, the v and h suffix is gone. Instead functions that expect a vector have the standard i suffix. And functions that accept both vectors and hashtables have no suffix.
Renamed functions that take a variadic number arguments, they got the suffix 'all' to distinguish them.

The reason for a separate set of functions even although the hashtable-variants would suffice, is simply performance.
Lua doesn't keep a count of the number of elements contained in a table. But, (binary) searching for it over a vector instead of doing whatever it is that unbound_search is doing, helps.
(http://www.lua.org/source/5.1/ltable.c.html#unbound_search)

The additional APIs at the moment, in alphabetical order: all, alli, any, anyi, count, counti, create, find, findi, join, joinall, joini, map, mapi, max, maxi, min, mini, retain, retaini, sum, sumi.

I'll include a complete API documentation once things stabilize. There is a small test suite included (lxsext_c_tests.lua).

Anyway, expect the game to crash using this.

Lastly, if there are any C/C++programmers interested in helping this project along, a GitHub repository is easily set up. Just sayin'.
 
Each word should be at least 3 characters long.
Search:    
Search conditions:    - spaces as AND    - spaces as OR   
 
Forum Index » S.T.A.L.K.E.R.: Clear Sky Forum » Mod discussion
 

All short dates are in Month-Day-Year format.


 

Copyright © 1995-2019 GSC Game World. All rights reserved.
This site is best viewed in Internet Explorer 4.xx and up and Javascript enabled. Webmaster.
Opera Software products are not supported.
If any problem concerning the site functioning under Opera Software appears apply
to Opera Software technical support service.