<< 11-06-2014 >>
00:01:38*superfunc joined #nimrod
00:10:37*Demos_ joined #nimrod
00:11:29*q66 quit (Quit: Leaving)
00:16:48*Skrylar joined #nimrod
00:31:34*Demos_ quit (Ping timeout: 276 seconds)
01:03:31*exetest joined #nimrod
01:04:46exetestirc client test
01:10:06*exetest quit (Remote host closed the connection)
01:19:21*DAddYE_ quit (Remote host closed the connection)
01:20:08*Jesin quit (Quit: Leaving)
01:21:12OrionPKnimrod irc client?
01:21:12OrionPK;)
01:22:29*hoverbear joined #nimrod
01:33:10*brson quit (Quit: leaving)
01:47:58*superfunc quit (Ping timeout: 246 seconds)
02:32:05*reloc0 quit (Ping timeout: 264 seconds)
03:03:40*bjz joined #nimrod
03:04:19*dymk quit (Ping timeout: 240 seconds)
03:09:04*def- joined #nimrod
03:12:11*def-_ quit (Ping timeout: 252 seconds)
03:12:26*xenagi quit (Quit: Leaving)
03:14:57Skrylarhrm
03:15:07Skrylari almost feel like writing a fuzzy logic module
03:25:20*freezerburnv quit (Quit: freezerburnv)
03:41:41*dymk joined #nimrod
03:47:29*BitPuffin quit (Ping timeout: 252 seconds)
03:53:04*flaviu1 quit (Ping timeout: 276 seconds)
03:56:14*XAMPP-8 joined #nimrod
04:01:58*reloc0 joined #nimrod
04:06:01VarriountSkrylar: If you do, don't forget to add the wooly thinking subroutines.
04:09:44*kemet joined #nimrod
04:46:43*BitPuffin joined #nimrod
05:09:46*XAMPP-8 quit (Ping timeout: 276 seconds)
05:11:46*XAMPP-8 joined #nimrod
05:15:04*XAMPP-8 quit (Read error: Connection reset by peer)
05:15:35*XAMPP-8 joined #nimrod
05:20:21*hoverbear quit ()
05:30:14*shodan45 quit (Quit: Konversation terminated!)
06:05:56*gsingh93_ joined #nimrod
06:23:19*zshazz_ joined #nimrod
06:25:53*zshazz quit (Ping timeout: 252 seconds)
06:26:20*XAMPP_8 joined #nimrod
06:26:25*XAMPP_8 quit (Read error: Connection reset by peer)
06:29:40*XAMPP-8 quit (Ping timeout: 260 seconds)
06:37:55*reactormonk_ quit (Ping timeout: 240 seconds)
06:40:45*zshazz__ joined #nimrod
06:44:35*zshazz_ quit (Ping timeout: 252 seconds)
07:03:18*bjz quit (Ping timeout: 240 seconds)
07:07:22*Puffin joined #nimrod
07:09:38*BitPuffin quit (Ping timeout: 240 seconds)
07:11:24*bjz joined #nimrod
07:28:15*zshazz__ quit (Quit: Leaving)
07:40:42*Puffin is now known as BitPuffin
08:04:51*kemet quit (Quit: Instantbird 1.5 -- http://www.instantbird.com)
08:12:18*XAMPP-8 joined #nimrod
08:12:49*XAMPP-8 quit (Read error: Connection reset by peer)
08:13:44*kunev joined #nimrod
08:26:16*joelmo joined #nimrod
08:26:48*io2 joined #nimrod
09:00:23*freezerburnv joined #nimrod
09:04:38*freezerburnv quit (Ping timeout: 240 seconds)
09:43:03*ehaliewicz quit (Ping timeout: 265 seconds)
10:09:10EXetoCOrionPK: yep
10:09:27*gsingh93_ quit (Quit: Connection closed for inactivity)
10:44:27*joelmo quit (Quit: Connection closed for inactivity)
11:10:39*jez0990_ joined #nimrod
11:12:46*eximiusw1staken joined #nimrod
11:14:27*jez0990 quit (Ping timeout: 265 seconds)
11:14:28*eximiuswastaken quit (Ping timeout: 265 seconds)
11:20:09*_dLog joined #nimrod
11:23:22*dLog_ quit (Ping timeout: 265 seconds)
11:24:40*kunev quit (Quit: leaving)
11:26:37*Skrylar quit (Ping timeout: 265 seconds)
11:26:45*Skrylar joined #nimrod
11:29:06*kunev joined #nimrod
12:12:42*BitPuffin quit (Ping timeout: 245 seconds)
12:17:42*untitaker quit (Ping timeout: 245 seconds)
12:19:18*BitPuffin joined #nimrod
12:24:31*untitaker joined #nimrod
12:33:30*flaviu1 joined #nimrod
12:51:27*CARAM_ quit (Ping timeout: 245 seconds)
12:53:06*CARAM_ joined #nimrod
13:05:12*Trixar_za quit (Ping timeout: 245 seconds)
13:05:19*Trixar_za joined #nimrod
13:19:09*darkf quit (Quit: Leaving)
13:34:43OrionPKEXetoC im working toward open sourcing IRC familiar, which is the irc client i wrote in nimrod
13:35:11OrionPK(which I've been using for the past several months)
13:48:19*BitPuffin quit (Ping timeout: 240 seconds)
14:04:12EXetoCOrionPK: client for android was it?
14:23:28*kunev quit (Quit: leaving)
14:28:01OrionPKEXetoC no, web client
14:28:15OrionPKEXetoC but it runs in android's version of chrome just fine
14:29:19EXetoCok
14:41:12*superfunc joined #nimrod
14:51:26*BitPuffin joined #nimrod
14:59:07*Araq0 joined #nimrod
15:00:00Araq0Hohoho
15:00:56Araq0Im bored. Somebody should entertain me
15:01:42EXetoCcan't think of another boring joke right now
15:03:42Araq0I only know german jokes and they tenf
15:04:38EXetoCquick, show him some gifs
15:04:44*Araq0 quit (Read error: Connection reset by peer)
15:04:51*AndChat-320025 joined #nimrod
15:07:23*Araq0 joined #nimrod
15:07:37*AndChat-320025 quit (Read error: Connection reset by peer)
15:11:36Araq0Dom96 !!!
15:12:17*BitPuffin quit (Ping timeout: 264 seconds)
15:15:06*superfunc quit (Quit: leaving)
15:15:45*superfunc joined #nimrod
15:24:50dom96Araq0: What do you want?
15:32:57*flaviu1 is now known as flaviu
15:36:00EXetoCwhat he said
15:53:05*Araq0 quit (Ping timeout: 264 seconds)
16:06:24*Jesin joined #nimrod
16:19:56*hoverbear joined #nimrod
16:38:01*brson joined #nimrod
16:45:47*Johz joined #nimrod
17:00:49*q66 joined #nimrod
17:00:49*q66 quit (Changing host)
17:00:49*q66 joined #nimrod
17:09:03*EXetoC quit (Quit: WeeChat 0.4.3)
17:12:39*DAddYE joined #nimrod
17:15:20*gsingh93_ joined #nimrod
17:16:07*EXetoC joined #nimrod
17:21:11*DAddYE_ joined #nimrod
17:21:56*DAddYE quit (Ping timeout: 252 seconds)
17:35:43*vendethiel joined #nimrod
17:35:53EXetoCI guess I should send acks or something to synchronize bi-directional thread communication
17:36:47*reactormonk_ joined #nimrod
17:41:23*Matthias247 joined #nimrod
17:44:51dom96EXetoC: what are you working on (again?)?
17:46:42*noam_ joined #nimrod
17:49:18*noam quit (Ping timeout: 240 seconds)
17:59:16*DAddYE joined #nimrod
17:59:20*DAddYE_ quit (Read error: Connection reset by peer)
18:05:40*joelmo joined #nimrod
18:07:17EXetoCdom96: irc client
18:07:29EXetoCbut additional threads complicate things for no good reason
18:07:42dom96async that shit
18:12:21*zshazz joined #nimrod
18:16:13EXetoCjust using spawn is fine
18:22:00EXetoCnot that async spawns threads yet
18:24:20dom96what are you using threads for?
18:26:24EXetoCdom96: for everything that isn't the UI
18:32:13dom96so sockets?
18:34:45*def-_ joined #nimrod
18:37:50*def- quit (Ping timeout: 252 seconds)
18:39:44EXetoCdom96: basically
18:45:19dom96so why not use async?
18:51:14EXetoCI don't know what problem it would solve in this case
18:52:24dom96it would be more efficient I think
18:54:49EXetoCthe overhead of parsing socket messages should be roughly nil anyway
18:56:47dom96That is something I should benchmark
18:56:52dom96async vs spawn sockets
18:58:57EXetoCwhen async gets support for threads?
19:00:01araqEXetoC: we're constantly thinking about it, but it's a very hard problem, on windows at least
19:00:31EXetoCok well I don't know what it is you're comparing otherwise
19:01:50araqwell async doesn't block, threading blocks but some other thread can run in the meantime
19:04:44Matthias247the main advantage of async vs. threads is you don't have to care about synchronization/locking
19:08:21EXetoCanyway, time for another rewrite
19:11:49dom96Perhaps we should just tell people to use blocking sockets + spawn :P
19:15:45EXetoCnot sure if srs :>
19:20:07*Jesin quit (Quit: Leaving)
19:28:48EXetoChow well are we doing with respect to minimizing allocations?
19:36:17EXetoCwe do have garbage collection so naturally there's no need to care in many cases
19:42:41def-_Hm, the example nimrod code on nimrod-lang.org is about twice as slow as equivalent python
19:43:23def-_Profiling says it's readLine()'s fault
19:48:29EXetoCwith an optimized build?
19:48:32def-_yes
19:48:44def-_also tried boehm gc but was even slower
19:49:07EXetoChow big is the input?
19:49:16def-_300 MB
19:49:43flaviudef-_: Try running it again
19:49:56def-_flaviu: i did a few times, it's not the cache
19:52:19flaviudef-_: Is there somewhere I can download the file?
19:52:39*ehaliewicz joined #nimrod
19:53:23EXetoChow long are the lines?
19:54:13EXetoCI don't know how python manages the buffers
19:54:13def-_Here's the code in python too just in case: https://gist.github.com/def-/45568077b01d2326a7bc
19:54:32def-_average line length ~73
19:56:10flaviuIts spending an awful lot of time in getc for me
19:56:15EXetoCok well there's a comment in readLine saying that it could be optimized a bit. I don't know if that means a little or a lot
19:56:44flaviuEXetoC: A lot. I'm sure theres a better way than byte by byte
19:56:48EXetoCflaviu: in total you mean? it has to call that for each char of ccourse
19:57:18def-_Ok I created a new file like this: dd if=/dev/urandom of=/media/log2 bs=1k count=300k
19:58:32EXetoCflaviu: is it possible with ansi C?
19:58:35flaviuEXetoC: I don't know C, but fgets() lets you specify a buffer size
20:00:49*Jehan_ joined #nimrod
20:03:09EXetoCyeah
20:03:54flaviuWhere did you find readline? Grep doesn't seem to turn up anything
20:04:27def-_https://github.com/Araq/Nimrod/blob/7a557013ae71761fd30c7330e104d73b959aeeda/lib/system/sysio.nim#L70
20:06:04flaviuYeah, that doesn't look efficient
20:08:16def-_This might be (part of) the python implementation, using a buffer of 8 KB by default: http://hg.python.org/cpython/file/3.2/Lib/fileinput.py
20:10:25Jehan_Did you compile with optimization? The readLine loop is not very efficient, but it still reads files faster for me than Ruby or Python.
20:11:02def-_I do nimrod -d:release c avg.nim
20:11:09Jehan_def-_: Normally, the FILE functions of ANSI C should have a buffer, so not necessary to replicate that if you use getc() and friends to begin with.
20:11:31flaviuJehan_: Python 3? I think it does fancy unicode stuff, so it might be slower.
20:11:41Jehan_flaviu: No, Python 2.
20:12:13Jehan_import fileinput
20:12:13Jehan_for line in fileinput.input():
20:12:13Jehan_ pass
20:12:52flaviuits seems like getc acquires a lock.
20:12:55def-_Jehan_: I posted my python code in the gist
20:13:18Jehan_def-_: I was trying to look at File I/O in isolation for this.
20:14:05flaviuhttp://i.imgur.com/ecyX3hw.png
20:14:28flaviuNo idea why else it would need atomic operations
20:16:49Jehan_def-_: On my machine, avg.nim seems to be about 3 times as fast as avg.py.
20:17:20Jehan_Running Python 2.7.6
20:18:07def-_Jehan_: with a file created from random?
20:18:31Jehan_def-_: Essentially pasting /usr/share/dict/words a few times.
20:18:35def-_because i have the same python version, and avg.nim is 4 times slower than avg.py
20:18:50Jehan_def-_: What OS?
20:18:58def-_ah, /usr/share/dict/words makes for shorter lines, so that may be it
20:19:34Jehan_def: Still shouldn't totally invert the relation.
20:19:58flaviudef-_: I can reproduce your problem
20:20:27Jehan_I'm running OS X. May it be different C stdlibs?
20:20:43def-_linux 64bit here
20:22:02flaviulinux 64 here too. Whats funny is that I can't even find gets in the profiler output for python
20:22:22def-_Jehan_: nope, it's the line length
20:22:49Jehan_Hmm, used longer lines, it's now about the same speed.
20:22:50def-_Jehan_: when i use /usr/share/dict/words (many times concatenated) avg.nim is indeed 3 times faster
20:23:05flaviuPython uses mchar, and it only takes 5% of total execution time.
20:23:12flaviu*memchar
20:23:35flaviuWait, thats to look for newlines
20:24:10flaviummap and memcpy it looks like maybe?
20:24:12Jehan_flaviu: I would think that memory allocation should create more of an overhead.
20:25:19flaviuMemory allocation is just 2.5% of total execution time
20:25:37*reactormonk_ quit (Ping timeout: 245 seconds)
20:25:42Jehan_flaviu: Compared to using memchr vs. comparing each character.
20:25:56Jehan_Manually, that is.
20:26:35araqJehan_: I've implemented a more "efficient" isObj for method dispatch but it sucks
20:26:44Jehan_Mind you, I don't disagree that readLine() would benefit from a do-over, but I'd like to see what the actual bottleneck is.
20:26:52Jehan_araq: Heh. :)
20:27:05araqI use a full type graph and compile 'of' into
20:27:16flaviuJehan_: Practically everything is the bottleneck
20:27:18Jehan_Ouchies.
20:27:47flaviuJehan_: http://i.imgur.com/eDi4BDE.png
20:27:53*hoverbea_ joined #nimrod
20:27:53araqif typeField == ojb or typeField == subObj1 or typeField == subObj2 etc.
20:28:14araqbut it grows quickly because TObject has lots of potential subtypes ...
20:28:26Jehan_flaviu: With inlining, it may be hard to tell.
20:28:41araqbut I now have a much better solution
20:29:37Jehan_araq: That would be?
20:30:03araqif x == subclass or x.base == subClass: return true # these 2 checks are fast and require no 'nil' checks
20:30:22araqelif x == blackListEntry: return false
20:30:32araqelif x == whiteListEntry: return true
20:30:41*hoverbear quit (Read error: Connection reset by peer)
20:30:51*hoverbear joined #nimrod
20:30:57araqelse: # slow linear search but set blackListEntry or whiteListEntry
20:31:32araqthe first 4 checks (perhaps only 2 or 3) are inlined
20:31:53*DAddYE quit (Read error: Connection reset by peer)
20:32:08araqthe codegen produces the *listEntries for 'isObj' to use
20:32:27araqsince it's only a single word stores, there are no threading issues
20:32:37*DAddYE joined #nimrod
20:32:42flaviuWell, the current bottleneck is getc, but looking at the assembler, the secondary bottleneck is the if statements checking for eol. So maximum bang-for-buck would be to replace getc with fgets and the if statements with memchar.
20:32:58Jehan_araq: Hmm. I know why I'm not a fan of multimethods. :)
20:33:42araqthe sweet part is the blacklist entry from which I hope to get some big gains in practice
20:33:42Jehan_flaviu: Replacing fgetc with getc doesn't change much for me.
20:33:57flaviufgetS
20:33:58flaviuhttp://www.cplusplus.com/reference/cstdio/fgets/
20:34:08*hoverbea_ quit (Ping timeout: 240 seconds)
20:34:31Jehan_araq: Still missing a bit of context. Is this still cache-based, i.e. trying to pack enough information inside a single word?
20:34:51Jehan_flaviu: I know what you said.
20:35:17Jehan_flaviu: I was talking about replacing fgetc() with getc(), which is easy and replaces a function call with a macro.
20:35:30Jehan_To get an estimate for how much of a bottleneck fgetc is.
20:36:01flaviuIt isn't, its getc that takes 59% of the time
20:36:11araqJehan_: I use a cache with 2 entries, yes
20:36:22Jehan_araq: So, making it thread-local then?
20:36:28flaviumaybe I'm missing something
20:36:47flaviuoh, I see. It uses fgetc in the source.
20:36:53araqJehan_: nope, it's global ("static" in C)
20:37:18flaviuThat gets optimized out into getc, it'd make no difference.
20:37:30Jehan_araq: Okay, if you have two words per entry, I'm not sure how that's going to work. At least not off-hand.
20:38:10araqJehan_: in a way we specialize isObj(x, ConstClassField) into isObjConstClassField(x)
20:38:27araqevery specialization has its own cache with 2 entries
20:39:06araqthe 2 entries have different semantics however, one slot says "computed and is NOT of type ConstClassField"
20:39:22araqthe other says "computed and IS of type ConstClassField"
20:39:47araqhence we always only store a single word
20:39:56*clone1018 quit (*.net *.split)
20:40:04*clone1018 joined #nimrod
20:41:18Jehan_flaviu: I think the takeaway is that somebody should optimize readLine(). The tricky part is still doing it portably and without changing the interface.
20:41:33araqso we can do: if x == cacheSlotT: return true
20:41:41araqif x == cacheSlotF: return false
20:41:47*reactormonk_ joined #nimrod
20:42:41araqdue to how type info is stored 2 other checks are cheap and shouldn't affect the cache, but that's an implementation detail
20:44:39EXetoCflaviu: you only grepped system.nim?
20:44:49Jehan_araq: Don't you still have to check that it's the correct method?
20:44:59Jehan_araq: Or are you simply caching the object graph?
20:45:10flaviuI grepped lib/, but I grepped with a export star, which turns out not to apply
20:45:30EXetoCok
20:45:33araqI simply cache knowledge about the object graph, Jehan_
20:46:02araqthe methods are already inlined / direct calls anyway
20:46:19araqthe problem is that isObj is not inlined afaict
20:46:56Jehan_araq: I see.
20:47:11Jehan_So you still have one check per subclass?
20:47:26araqunfortunately, yes
20:47:38Jehan_araq: That's not going to fix anything.
20:48:12Jehan_Well, it does improve on the current state of affairs.
20:48:21Jehan_But still linear in the number of subclasses.
20:48:46Jehan_So, not a long-term solution.
20:48:53araqyeah well
20:49:06araqC really has a huge advantage here
20:49:20Jehan_C?
20:49:25Jehan_Do you mean C++?
20:49:28araqdispatchTable[i + j*c]()
20:49:31araqI mean C
20:49:57araqC makes you do it by hand hence you can easily use small integers instead of type info pointers
20:50:04Jehan_Well, if you knew the indexes at compile time, you could do that too.
20:50:15araqyeah but this is harder than it looks
20:50:41*reactormonk_ quit (Ping timeout: 264 seconds)
20:50:42araqplus in C I have no subtyping really, so I can base it on == instead of <= (subtyping)
20:51:37araqwhich is why I want to get rid of methods and let the programmer do dispatchTable[i + j*c]() via some clever macros
20:52:02Jehan_In other words, you can make things a lot faster by giving up most of what actually makes doing it worthwhile. :)
20:53:11araqI disagree. IMO the subtyping often simply sucks for dispatching and I'd rather have == instead of <= semantics
20:54:33Jehan_araq: The point is usually abstraction.
20:55:51Jehan_Another big problem is that multimethods are a lot harder to deal with than single dispatch.
20:56:44*Jesin joined #nimrod
20:57:50Jehan_Which reminds me that with multi-methods you'll eventually also need something like next-method/call-next-method.
20:58:02Jehan_Which is also a fairly big headache to implement.
21:00:46araqI don't remember what these do :P my Lisp is too rusty
21:01:15Jehan_It's basically the multi-method equivalent to super calls.
21:01:28Jehan_Dispatching to the next less specific method.
21:01:41Jehan_Not really an equivalent, but a lot of overlapping use cases.
21:01:50araqyeah that's what I would have guessed
21:02:07flaviuSomething like stackable traits in scala?
21:02:22araqoh well, maybe I shouldn't bother
21:02:32Jehan_Hmm.
21:02:59Jehan_flaviu: No, unrelated. It's a multi-method thing.
21:03:03*Mat3 joined #nimrod
21:03:06Mat3hello
21:03:49araqhi Mat3
21:03:55Mat3hi araq
21:04:16araqJehan_: well we can always transform the linear search into a binary search at least
21:04:18Jehan_araq: If I had to pick something, I'd go with a thread-local cache right now.
21:04:31araqbut what do you cache?
21:04:47Jehan_method name + type signature.
21:04:51Jehan_It's what we do in GAP.
21:05:41araqthe method name then transforms a static call into a dynamic call
21:05:42Jehan_In the long term, I'd generate the dispatch table at link time.
21:06:09araqwe already have "link time code generation" for the dispatcher
21:06:11Jehan_Well, quasi-link-time.
21:06:28Jehan_Oh? That should kinda sorta solve the problem with dynamic libraries then?
21:07:39araqwell it's done before C compiler invokation
21:07:50araqso no, DLLs still are a problem
21:10:28Jehan_That would be assuming that type/method numbering remains consistent across multiple DLL versions.
21:11:04Jehan_Incidentally, binary search can be reasonably efficient if inlined and optimized (see SmallEiffel/SmartEiffel).
21:11:32Jehan_My guess is that it works reasonably well with branch prediction as long as dispatch isn't entirely random.
21:12:35fowlsomeone should fix ffi for the vm
21:13:07EXetoCweren't we going to drop support for it?
21:13:11EXetoCfowl: what do you want to do?
21:15:51fowluse the os module at compiletime
21:18:28*Roin quit (*.net *.split)
21:19:14*Roin joined #nimrod
21:19:24araqfowl: that's planned but won't happen via the FFI
21:20:27Jehan_araq: I'd look at giving type descriptors a numeric id field.
21:20:51Jehan_araq: This way, you can give types in DLLs the same numbering regardless of how they are compiled.
21:21:31araqyeah but then you actually use a hash table string->int at program startup
21:22:31araqso you can easily do 'object of mylib.foo.someObj' and then produce efficient small integers at runtime
21:22:54Jehan_araq: Not sure -- each type in the DLL that is exposed externally would export a C symbol (at worst, a function) that can be used to initialize the id.
21:23:38Jehan_araq: Mind you, I don't know how viable that would be with the current compiler.
21:23:49araqhmm yeah that's another solution
21:24:20Jehan_Of course, technically that's another table lookup, except that the dynamic linker does it (and can optimize it). :)
21:25:09Jehan_I'd be more concerned that this doesn't scale.
21:25:28Jehan_E.g., App uses DLL1 and DLL2, both DLL1 and DLL2 use DLL3.
21:25:42Jehan_So DLL1 and DLL2 are likely to generate different type numberings for DLL3.
21:30:11Jehan_Can be solved by double indirection, but at this point we're starting to look at a fair amount of overhead.
21:30:46Jehan_It would probably still beat a hash table.
21:32:14Jehan_switch (remote_type_id_to_local_type_id[remote_type_id_foo_xxxx]) { … }
21:32:46araqwell SmartEiffel never supported DLLs as far as I know
21:32:57Jehan_remote_type_id_to_local_type_id[] would essentially be a dynamically sized array.
21:33:21Jehan_araq: No, DLLs are hard in a number of ways if you're looking at something that the loader doesn't support out of the box.
21:34:07*Johz quit (Ping timeout: 272 seconds)
21:34:15Mat3get some sleep, ciao
21:34:24*Mat3 quit (Quit: Verlassend)
21:34:25Jehan_Like thread-local variables for lightweight threads.
21:35:12fowlanyone want to help debug my app
21:36:51Jehan_Speaking of which, the absence of thread-local storage is another fatal flaw in Go (at least for my purposes).
21:40:58araqwell Rob Pike doesn't need TLS so it must be a bad idea :P
21:41:14araqugh this new Linux Mint is unstable like Win95
21:41:26araqrandom hard crashes
21:41:41araqhow am I supposed to find the issue?
21:42:05araqlook at /etc/banana/log/crashreport ?
21:50:01def-_Jehan_, flaviu: I have a faster version using fgets now: https://gist.github.com/def-/45568077b01d2326a7bc
21:50:26def-_But I didn't get the \l\r behaviour of readLine replicated
21:50:45def-_and it uses a global buffer, not sure how to do that properly
21:52:21Jehan_def-_: If you're using an iterator, you can put it in the iterator's local variable.
21:53:44araqa global buffer is not acceptable by any means
21:53:52araqhowever
21:54:15araqthe current implementation is complex because I wanted a working Eof()
21:54:52araqI think we should drop this idea and then we can make it efficient more easily
21:55:24araqthe alternative means we need to do our own buffer handling but then TFile is not C's FILE* anymore
21:55:36araqwhich is too useful for C interop
21:55:42Jehan_Looking at the code, you could easily make the buffer local, too?
21:56:09Jehan_def-_: There's no reason why you can't do a local "var buf: array[8192, char]"
21:56:13def-_Jehan_: local in the function or the caller?
21:56:21Jehan_Local in the function.
21:56:28def-_then it's a lot slower
21:56:32Jehan_Why?
21:56:32def-_i think
21:56:44Jehan_array[8192,char] is allocated on the stack.
21:56:59Jehan_This isn't Java.
21:57:14Jehan_It's more like a char buf[8192] in C.
21:59:34def-_But with an array I can't use len
22:01:11def-_Ok, got it working, but it's still a lot slower
22:02:11Jehan_proc c_strlen(s: cstring) {.importc, header:"<string.h>", noSideEffect.}
22:02:26Jehan_Hmm, wondering if that isn't exposed somewhere for array of char already.
22:03:22araqJehan_: how does a local buffer work?
22:03:39araqyou read too much and then it's discarded
22:03:46Jehan_araq: He's only using a buffer as an argument to fgets.
22:03:57def-_Yeah, the buffer never reads too much
22:04:03araqoh I see
22:04:20def-_So it should work with a local one, but it takes 2 times as long
22:04:36Jehan_Let me look at it.
22:04:44araqwell fgets/lines is a quite stupid interface to begin with
22:05:10araqwhat we really need is readLineBlock
22:05:35araqso it returns a block but that's guaranteed to end in a newline
22:05:49def-_Jehan_: it's localavg.nim in the gist
22:06:10araqthen you sometimes need to splitLines on the block
22:06:11def-_oh wait, that looks bad
22:06:39araqbut it's a very good idea to do it this way for parallel processing
22:08:37Jehan_def-_: You don't need the extra cstring variable. You can pass the array to fgets directly.
22:08:47Jehan_def-_: That said, for some odd reason it's slower for me, too.
22:08:47def-_Jehan_: yeah, just fixed
22:08:50def-_doesn't change anything
22:09:04Jehan_In practice, it just targets the stack rather than the heap.
22:09:12Jehan_But otherwise, the code should be identical.
22:09:43Jehan_Oops, forgot the {.noinit.}
22:10:35def-_Jehan_: that's cool, now it's just as fast as the global one
22:10:45araqhey no comments on my readLinesBlock idea?
22:11:01def-_araq: I'm not sure I understood it correctly
22:11:19Jehan_araq: Hmm, I must have missed it?
22:11:37def-_araq: It can read multiple line into a block and then we need to parse the block for newlines again?
22:11:42Jehan_araq: It may be useful to have a BufferedFile type in addition to TFile.
22:12:12araqfor line in lines("foo.txt"): spawn processLine(line) # argh, slow
22:12:25Jehan_def-_: It should be faster. Try: for line in stdin.readAll.splitLines: ...
22:12:40araqfor line in linesBlock("foo.txt"): spawn processLine(line) # yay, fast
22:13:13araqprocessLine then needs to do slightly more work as 'line' is in fact multiple lines
22:13:23Jehan_araq: I'll go out on a limb and say that this is not ever going to be fast. :)
22:13:24araqbut usually that's really simple to do
22:13:42araqJehan_: sure it is, why wouldn't it?
22:13:51Jehan_The synchronization overhead is going to kill you, unless processLine() does a lot of work.
22:14:12araqwell yes, that's what I'm proposing
22:14:35Jehan_Yeah, but how much code is really there where processLine would be expensive?
22:14:53araqdepends on everything of course :P
22:15:03*noam_ is now known as noam
22:15:44araqbut you're right there is a better way to do the same
22:15:48flaviuJehan_: Not much experience here, but wouldn't SIMD help?
22:16:18Jehan_flaviu: We're processing character strings of unknown length here?
22:16:44flaviuWell, we know it'll be at least a certain size (8192)
22:16:47Jehan_SIMD is really for where you do (mostly) identical stuff on identical-sized data.
22:17:40Jehan_flaviu: Not sure how that would help.
22:18:52*saml_ joined #nimrod
22:19:25flaviuI had an idea, but its impractical after thinking about it a bit more
22:25:22Jehan_flaviu: In the end, processing files line by line is not one of the typical HPC challenges that I'd worry about.
22:25:58Jehan_Avoiding unnecessary overhead is one thing, but this is O(n) territory still.
22:26:19Jehan_For anything sufficiently large, you'll be I/O-bound, anyway.
22:27:02flaviuI guess using strchr would be fast enough anyways, even if its 2x O(n)
22:27:33Jehan_strchr() wouldn't work if the file has zeroes, anyway.
22:27:44flaviuyou're right, memchr
22:28:22flaviuWith some heuristics, you could probably avoid running 2x memchr 99% of the time too.
22:29:32araqJehan_: processing files line by line is a very common use case
22:29:46araqand can usually be sped up with some parallelism
22:30:18araqit's not a *hard* interesting problem, but very common
22:31:39flaviuI think its more important for the code not to be too clever, especially in the stdlib.
22:38:03*io2 quit (Quit: ...take irc away, what are you? genius, billionaire, playboy, philanthropist)
22:43:28Jehan_araq: How do you avoid this becoming I/O bound?
22:44:26Jehan_flaviu: Agreed. If you want to do fancy stuff, then a specialized library is best.
22:44:55araqflaviu: well the current readline is too slow and many people noticed. what do you suggest?
22:46:39Jehan_araq: Improving upon the current version isn't the same as being extra fancy. :)
22:47:38*vendethiel quit (Ping timeout: 240 seconds)
22:47:40Jehan_You want the basic readLine implementation to be robust, too.
22:47:51araqJehan_: again, I refer you to this: http://www.1024cores.net/home/scalable-architecture/parallel-disk-io
22:48:18araqparallelism is useful for IO bound things too
22:49:08*vendethiel joined #nimrod
22:49:43Jehan_araq: For stuff like this, I'd subdivide the file in fairly large segments, not lines.
22:50:08Jehan_Then assign segments to threads.
22:50:12araqer that's what I meant
22:50:30araqbut usually you need the line boundaries for correctness
22:50:37araqhence my "line blocks"
22:50:51Jehan_araq: Scan ahead to the next newline.
22:50:57Jehan_After doing a seek.
22:51:14araqyes, exactly
22:51:26araqor scan back to the last newline
22:51:35araqand add the overlap back to the buffer
22:52:22Jehan_araq: Which doesn't matter if you kill performance with spawning threads for each line.
22:53:03araqwhich is what I meant
22:59:01*zshazz quit (Quit: Leaving)
23:02:02flaviufgets and memchr seems to be the best solution
23:02:36flaviuThat way, its the stdlib's problem to figure out a way to do things effectively, which might depend on the platform you're on.
23:03:05araqflaviu: ok, but I'll add the linesBlock proc anyway
23:04:30*reactormonk joined #nimrod
23:04:46araqflaviu: how do I find the cause for random crashes on linux?
23:05:13flaviuWhat is crashing? The whole computer?
23:05:24araqyep
23:05:38araqfreezes, not a single key works
23:06:35def-_araq: does magic sysrq work?
23:06:47flaviuI guess look at the kernel logs and then the xorg logs?
23:07:09*Matthias247 quit (Read error: Connection reset by peer)
23:07:10flaviuApparently ubuntu has a log viewer: http://askubuntu.com/questions/15647/where-can-i-find-the-kernel-logs , linux mint might have one too
23:09:36araqdef-_: what's the magic sysrq?
23:10:23def-_key combinations directly to the kernel, alt + sysrq + some key
23:11:08araqI only know about ctrl+alt+backspace and that didn't work
23:11:29flaviuthat key combination goes into xorg IIRC
23:11:59def-_might be an xorg problem then. can you ping/ssh on the computer when it's unresponsive?
23:14:13araqdunno
23:14:51flaviuDid the logs have anything interesting?
23:16:18araqdunno where to find X11's logs
23:16:37def-_/var/log/Xorg.*
23:17:18araqwell I'm searching for "fatal" and "error"
23:17:35flaviuYou can also try searching the start menu in linux mint for "log" to find the log viewer.
23:17:50flaviuIt might be better to start from the end and manually scan up
23:19:11araqthe logs read like somebody enjoying intellectual masturbation
23:21:12araqthe only error that I can see is: [ 26.487] (EE) Failed to load module "nv" (module does not exist, 0)
23:21:39araqmaybe I should install the real nvidia driver and hope for the best
23:21:56flaviuThat's harmless
23:22:19*Boscop quit (Read error: Connection reset by peer)
23:22:38flaviuIt happened 26ms or 26s after X started, I doubt you saw freezing at that time
23:22:39*Boscop joined #nimrod
23:23:08flaviuBut installing the blob driver might give you better performance for some things, so its worth trying out
23:23:29araqflaviu: yeah I know
23:23:44araqbut then I can't fix it by doing nothing either
23:29:21*hoverbear quit ()
23:31:00def-_I'll try to get some new Nimrod users with a talk (in German) in a week: https://entropia.de/GPN14:Fahrplan
23:31:31flaviuAwesome, a secondary source!
23:33:10dom96def-_: awesome, will the talk be recorded?
23:33:14Jehan_flaviu: Don't hold your breath. :)
23:33:23def-_dom96: yes
23:33:36def-_there's even a livestream i think
23:33:54dom96ooh, it may just be a viable secondary source then
23:34:07def-_but it will be in German =/
23:34:09Jehan_flaviu: I suspect that there's still a bit of resistance to be overcome w.r.t. Wikipedia politics to establish notability.
23:34:17dom96language doesn't matter
23:34:22dom96AFAIK
23:34:42Jehan_dom96: No, language doesn't matter.
23:34:45*darkf joined #nimrod
23:34:57dom96bigger problem is them not trusting the source
23:34:57flaviuYou'd probably need an English translation though
23:35:16Jehan_flaviu: It's not for dom96, it's for Wikipedia.
23:35:46dom96There is plenty of Wikipedia people who can speak German.
23:35:53flaviuOk, sounds good then
23:36:03dom96We simply need this to establish notability.
23:36:08Jehan_dom96: Exactly. And, unfortunately, who can be very stereotypically German. :)
23:36:24dom96hehe
23:36:27Jehan_I.e. obsessed with rules and literalness. :)
23:54:08*brson quit (Quit: leaving)