<< 18-02-2019 >>
00:00:09*rnrwashere joined #nim
00:12:52*shodan45 quit (Quit: No Ping reply in 180 seconds.)
00:17:18*shodan45 joined #nim
00:20:46*noeontheend joined #nim
00:39:09FromGitter<irskep> I've got some code that compiles in one file and doesn't compile in another. Could really use some help if anyone's up for it. Details to follow.
00:39:24FromGitter<irskep> This compiles: https://github.com/irskep/nimrl/blob/master/src/scenes/state_debug_scene.nim#L24
00:39:28FromGitter<irskep> This does not compile: https://github.com/irskep/nimrl/blob/master/src/input.nim#L7
00:40:20FromGitter<irskep> The procs are identical. `IsKeyPressed` is defined by `raynim`, which is a nimterop-based wrapper of raylib (https://www.raylib.com/). `KEY_UP` is an num of some kind, and seems to be where the trouble is, but I can't reason about what the trouble would be, exactly.
00:43:12FromGitter<irskep> My best guess is that the `KeyboardKey` enum is missing its underlying `cint` data type somehow, but that seems like very surprising behavior
00:44:37FromGitter<irskep> Solved it, PEBKAC: I wrote a wrapper for `IsKeyPressed` that does the type cast and forgot to import it.
01:05:02*stefanos82 quit (Remote host closed the connection)
01:09:58FromGitter<dawkot> Hi, what is the preferred way to get a random element of a `HashSet`? Just converting it to a seq and using the `sample` method from `random` on it?
01:25:12FromGitter<irskep> I'm not aware of any standard library that gives you a nice way to randomly sample something that isn't a linear list. :-) Not being an expert, that seems like it'd be reasonable.
01:27:13FromGitter<irskep> Even the very latest version of Python doesn't have anything like that, and they have a kitchen sink stdlib
01:27:30*Cthalupa quit (Read error: Connection reset by peer)
01:28:12FromGitter<dawkot> I guess that means you're right
01:28:41FromGitter<dawkot> Never needed random sampling before so I didn't get to know what other languages offer
01:40:24*Cthalupa joined #nim
01:58:20*fanta7531 joined #nim
02:18:35*Tyresc quit (Quit: WeeChat 2.4-dev)
02:23:52*fanta7531 quit (Quit: fanta7531)
02:25:43*ng0_ joined #nim
02:28:29*ng0 quit (Ping timeout: 256 seconds)
02:55:04*rnrwashere quit (Remote host closed the connection)
02:58:21*rnrwashere joined #nim
03:02:31*banc quit (Quit: Bye)
03:15:30*dddddd quit (Remote host closed the connection)
03:23:34*banc joined #nim
03:59:28shashlick@irskep: how is nimterop working for you
04:00:47*Cthalupa quit (Ping timeout: 255 seconds)
04:03:27*nsf joined #nim
04:49:34*noeontheend quit (Ping timeout: 258 seconds)
04:51:24*noeontheend joined #nim
05:00:10*chimez joined #nim
05:04:02*Snircle quit (Quit: Textual IRC Client: www.textualapp.com)
05:18:38*noeontheend quit (Ping timeout: 255 seconds)
05:22:34*chimez quit (Quit: chimez)
05:23:15*Ven`` joined #nim
05:26:10*narimiran joined #nim
05:28:02*Ven`` quit (Ping timeout: 272 seconds)
05:29:12*rnrwashere quit (Remote host closed the connection)
05:34:09*rnrwashere joined #nim
05:35:01*rnrwashere quit (Remote host closed the connection)
05:35:15*rnrwashere joined #nim
05:35:23*chimez joined #nim
06:11:27*rnrwashere quit (Remote host closed the connection)
06:17:46*Vladar joined #nim
06:35:48*chimez quit (Quit: chimez)
07:01:47*krux02 joined #nim
07:18:44*vlad1777d joined #nim
07:27:50*chimez joined #nim
07:28:01*chimez quit (Client Quit)
07:28:14*vlad1777d quit (Ping timeout: 255 seconds)
07:32:58FromGitter<Varriount> Dawkot: Generate a random number x, and perform x iterations through the hash keys?
07:45:20*PMunch joined #nim
07:54:34*vlad1777d joined #nim
08:00:00*gmpreussner quit (Quit: kthxbye)
08:04:37*gmpreussner joined #nim
08:21:31*ng0_ quit (Ping timeout: 256 seconds)
08:25:57*ng0_ joined #nim
08:36:54FromGitter<mratsim> other mods, is that an elaborate troll or a genuine thread? https://forum.nim-lang.org/t/4664 cc @dom96, @Araq?
08:37:21PMunchHaha, I can only see the title but it smells troll
08:38:03FromGitter<mratsim> @dawkot I build my own "set" with sampling for my game of go needs
08:38:08*chimez joined #nim
08:38:30FromGitter<mratsim> https://github.com/mratsim/golem-prime/blob/master/src/datatypes.nim#L106-L119
08:38:36narimiran@mratsim i've seen that and decided not to make it public
08:39:14narimiranlook at this: "The instructions say "build something? . We don't have Home Depot or Lowes around here"
08:39:57FromGitter<mratsim> and random sampling is like this: https://github.com/mratsim/golem-prime/blob/master/src/move_heuristics/m_move.nim#L19-L23
08:42:24*ng0_ quit (Quit: Alexa, when is the end of world?)
08:42:48FromGitter<mratsim> if you want multinomial sampling (i.e. sampling with different probability distribution), it was added in devel, implementation using cumulative sum + binary search. Here are 2 implementations that I'm currently evaluating: https://github.com/numforge/laser/blob/master/benchmarks/random_sampling/bench_multinomial_samplers.nim (1 is cumsum + binary search and the other is based on Binary Idnexed Tree (Fenwick Trees))
08:56:26*vlad1777d quit (Ping timeout: 255 seconds)
09:15:11*abm joined #nim
09:18:30*floppydh joined #nim
09:30:52*neceve joined #nim
09:34:00*neceve quit (Client Quit)
09:34:25*neceve joined #nim
09:36:41*skelett quit (Quit: WeeChat 2.1)
09:37:05*skelett joined #nim
09:54:38*PMunch quit (Ping timeout: 245 seconds)
10:01:50*vlad1777d joined #nim
10:06:46*dom96_w joined #nim
10:12:10*rnrwashere joined #nim
10:16:52*rnrwashere quit (Ping timeout: 264 seconds)
10:19:00*PMunch joined #nim
10:33:47*lritter joined #nim
10:38:47krux02mratsim that forum post you linked to. I cannot tell if it is a troll. If it isn't, it is a very low level beginner, so low that teaching him would cost a lot of time and effort.
10:45:22Araqbeginner on Windows that doesn't understand the command line
10:45:44Araqand does not have English as his major tongue. I don't think he is a troll
10:57:16*chimez quit (Quit: chimez)
11:02:33FromDiscord<exelotl> code::blocks self traumatisation sounds about right
11:03:36*sealmove joined #nim
11:40:25sealmoveIs the code of Krux's project (the Nim -> GLSL) available online?
11:41:00*kapil____ joined #nim
11:41:37narimiransealmove: https://github.com/krux02/opengl-sandbox ?
11:41:50sealmoveoh, that's the repo, thanks
11:55:11sealmovekrux02: do you have a moment?
11:57:49sealmovekrux02: it's not really related to your project, but I thought you could give me some insight on using Nim macros to generate code of some other language.
11:59:54FromGitter<mratsim> you don't need macros, you can use templates for starter
12:00:19FromGitter<mratsim> it's probably easier to start with that
12:01:21FromGitter<mratsim> for example: https://flyx.github.io/emerald/tutorial.html
12:01:27FromGitter<mratsim> (to generate HTML)
12:01:46FromGitter<mratsim> ah sorry it might use macros
12:01:47sealmoveyeah, i wanted a simple example, thanks
12:02:19FromGitter<mratsim> this generates OpenCL: https://github.com/mratsim/Arraymancer/blob/master/src/tensor/private/p_kernels_interface_opencl.nim
12:02:53FromGitter<mratsim> for example this is self contained: https://github.com/mratsim/Arraymancer/blob/master/src/tensor/private/p_kernels_interface_opencl.nim#L128-L159
12:03:45FromGitter<mratsim> it generates an opencl function called "kern_name" with type "ctype" and the function is "op" and can be + - / * ...
12:03:55FromGitter<mratsim> it's just string interpolation
12:05:17FromGitter<mratsim> final usage: https://github.com/mratsim/Arraymancer/blob/master/src/tensor/operators_blas_l1_opencl.nim
12:05:40FromGitter<mratsim> at the bottom
12:05:42sealmoveAnother question: Nim DSL can't use arbitary syntax right? If I want a DSL that uses (almost) arbitary syntax, what's my best bet? Maybe I could make a simple parser which then connects the new language with Nim macros?
12:06:10FromGitter<mratsim> you can use anything that can be parsed by Nim parser
12:06:24FromGitter<mratsim> i.e. if you want to use "-" in an identifier you can't
12:06:56sealmoveI want to do Nim -> Bash :D
12:07:06FromGitter<mratsim> but the best way to check is to have a macro "macro myparser(body: untyped): untyped = echo body.treerepr"
12:07:32*dddddd joined #nim
12:07:52FromGitter<mratsim> and call it via: ⏎ ⏎ ```myParser: ⏎ mylist() ⏎ ofweirdStatement = IwantToTest``` [https://gitter.im/nim-lang/Nim?at=5c6aa018ecef85660bc684cb]
12:08:08FromGitter<mratsim> and you'll see if you get a parse error or not
12:09:17FromGitter<mratsim> if you want arbitrary syntax you need a custom tokenizer yes.
12:10:38FromGitter<mratsim> but the main limitations I'd say are postfix operators and stuff that are prevented in identifiers like "-" or starting a name with a "_" or having double "__"
12:11:42sealmovewhat about normal nim code? for example translating an if statement in nim to an if statement in bash?
12:13:55FromGitter<mratsim> you need to implement a code generator
12:14:49sealmovewhich takes Nim AST as input?
12:16:37FromGitter<mratsim> or Nim intermediate representation (whatever that is), have a look into Nim C generator: https://github.com/nim-lang/Nim/blob/devel/compiler/cgen.nim
12:16:51FromGitter<mratsim> or jsgen: https://github.com/nim-lang/Nim/blob/devel/compiler/jsgen.nim
12:17:13FromGitter<mratsim> or LLVM IR generator: https://github.com/arnetheduck/nlvm/blob/master/nlvm/llgen.nim
12:17:18FromGitter<mratsim> using Nim compiler as a plugin
12:17:43sealmoveawesome, thanks
12:18:30FromGitter<mratsim> might be relevant as well ;) https://github.com/NateGraff/blessedvirginmary
12:18:39FromGitter<mratsim> LLVM IR to Bash
12:20:04*nsf quit (Quit: WeeChat 2.3)
12:20:50FromGitter<mratsim> also you can check what was in the undocumented PHP backend: https://github.com/nim-lang/Nim/pull/7606
12:21:40sealmovePHP backend! Hadn't heard about that!!
12:22:51FromGitter<mratsim> it was deleted :P
12:23:30FromGitter<mratsim> basically it seems like it used a combination of emit + asm to produce a .php script
12:24:44*ng0 joined #nim
12:30:22federico3urgh
12:31:24sealmoveI know it's all in the documentation, but I don't get the rough idea of how to use Nim parser to specify a custom syntax :|
12:32:50federico3https://github.com/nim-lang/Nim/pull/7606/files how was it working tho...?
12:36:49FromGitter<mratsim> with untyped, Nim parser will gice you an abstract syntax tree
12:37:17FromGitter<mratsim> but only if it's something that is valid Nim parsing wise
12:37:37FromGitter<mratsim> if you want a custom syntax you need your own lexer/parser
12:37:52Araqfederico3: it assumed PHP is a variant of JavaScript :-)
12:38:37Araqworked quite well but we don't want to maintain it
12:38:48Araq„Thus a program analysis performing bounds checking for C arrays or strings,
12:38:48Araqsay, must account for such implicit pointer arithmetic. We therefore set out by
12:38:48Araqasking the following question: How much pointer arithmetic can one include in
12:38:49sealmoveoh I see! so I get some AST and I just observe it with a pretty printer and manipulate it
12:38:50Araqseparation logic and remain within polynomial time?
12:38:52AraqUnfortunately, and perhaps surprisingly, the answer turns out to be: essentially none at all.“
12:39:11FromGitter<mratsim> but I'm having trouble to understand what you want, on one hand you want Nim -> bash, and on another you want custom syntax (so not Nim) to --> ??? conversion
12:39:43FromGitter<mratsim> @sealmove yes
12:40:03sealmovemratsim: I think Nim's syntax will cover it, probably won't need a custom one
12:40:16FromGitter<mratsim> and the pretty printer is echo myAST.treerepr
12:43:33sealmoveSo basically all I need is the code generator, and Nim -> bash direct mapping seems possible
12:44:51sealmoveJust wondering if I am missing something
12:45:01FromGitter<mratsim> best is to try :P
12:45:11sealmoveyeah, will get my hands dirty
12:45:13FromGitter<mratsim> I'm writing my own compiler and I'm doing things step by steps
12:45:27sealmoveoh, what compiler is it?
12:45:36FromGitter<mratsim> for example, after 7 steps of front-end refinements, this is my proof of concept: https://github.com/mratsim/compute-graph-optim/blob/master/e08_DSL_compiler.nim
12:45:49FromGitter<mratsim> Deep Learning and Image processing compiler
12:46:00sealmoveiGo!
12:46:14FromGitter<mratsim> here it only supports expressions
12:46:28FromGitter<mratsim> from yesterday, i support statements and in-place updates: https://github.com/mratsim/compute-graph-optim/blob/master/e09_DSL_compiler_stmt.nim
12:46:47FromGitter<mratsim> next step is to support arrays and not just scalars
12:47:02FromGitter<mratsim> then during codegen, generates SIMD instructions
12:47:10FromGitter<mratsim> then support LLVM IR codegen
12:47:22FromGitter<mratsim> then add runtime generation
12:47:33FromGitter<mratsim> then support GPU code gen
12:47:51sealmoveso custom language to do vector programming?
12:47:53FromGitter<mratsim> but don't do everything at once, start with an end to end simple example and starts adding features on top
12:48:19FromGitter<mratsim> the story is here: https://github.com/mratsim/compute-graph-optim/blob/master/arraymancer_discussion_347.md
12:48:51sealmovewill read
12:49:10FromGitter<mratsim> I'm refactoring the backend of Arraymancer (https://github.com/mratsim/Arraymancer)
12:50:09sealmovegreat! seems important, gl
12:50:17FromGitter<mratsim> and while working on the new backend (https://github.com/numforge/laser) I'm limited in various ways when combining various "macro block" like convolution and matrix multiplication
12:50:36FromGitter<mratsim> I'm doing runtime check of simd size everywhere
12:51:04FromGitter<mratsim> I need to do the same thing (vectorization, loop blocking, parallelization) often but in different ways.
12:51:56FromGitter<mratsim> and also while I already provide an imperative way to do loop-fusion, you have to everything in that single for loop so it'snot composable
12:52:19sealmovewhen it's finished, will you use it to make a iGo AI? :D
12:53:10FromGitter<mratsim> I already started one: but I'm stuck because of static/ranges/distinct types compiler issues - https://github.com/mratsim/golem-prime
12:53:26FromGitter<mratsim> the neural network part can already be done today.
12:53:40sealmoveI knew it!
12:54:04FromGitter<mratsim> the monte carlo tree search is actually the tricky thing, due to GC overhead in tree data structure, I have to do bench to make sure it's not too costly
12:54:19sealmoveyeah, have to wait for Nim v2 for that
12:54:43sealmove(new runtime thing)
12:54:44FromGitter<mratsim> here it's promising: https://github.com/frol/completely-unscientific-benchmarks
12:55:16FromGitter<mratsim> but a 19x19 game board is 300+ bytes not just 16
12:55:24FromGitter<mratsim> so the GC might collect much more often
12:55:30FromGitter<mratsim> so I would need a memory pool
12:55:35FromGitter<mratsim> so yeah
12:56:34sealmovehave you tried Shogi/
12:56:41FromGitter<mratsim> yes
12:56:46sealmove(btw minimal example of my idea https://bpaste.net/show/56c9b18f9bde)
12:57:04sealmoveI've been playing Shogi for ~10 years
12:57:16*ng0 quit (Quit: Alexa, when is the end of world?)
12:57:34sealmoveiGo unfortunately basically tried
12:57:46sealmovebarely*
12:58:04FromGitter<mratsim> I actually have this book at home: http://shop.nekomado.com/products/detail.php?product_id=38 (The Art of Shogi by Hopking)
12:58:23FromGitter<mratsim> but I didn't play that much
12:59:17sealmove am thrilled that someone from Nim has played it, even if a little
12:59:25FromGitter<mratsim> for your example, it might be easier to just use a jsonNode to bash command generator
12:59:41sealmovejsonNode, mmm
12:59:59FromGitter<mratsim> this way you can use json as config file
13:00:27sealmovei guess it's a good idea since json is popular
13:01:37FromGitter<mratsim> we're building something in that vein at status: https://github.com/status-im/nim-confutils
13:02:43FromGitter<mratsim> and you can use exec &"{conf.command} {conf.parameters}"
13:03:07FromGitter<mratsim> the {} interpolation comes from https://nim-lang.org/docs/strformat.html
13:03:47FromGitter<mratsim> for execution, seems like execProcess is your tool: https://nim-lang.org/docs/osproc.html#execProcess%2Cstring%2CopenArray[string]%2CStringTableRef%2Cset[ProcessOption]
13:04:08FromGitter<mratsim> i.e. AST is overkill for your need if it's a simple declarative thing
13:04:22FromGitter<mratsim> not AST / codegen
13:04:48sealmovewell, i was hoping for a complete Nim -> Bash implementation. translate not only commands (like my minimal example) but also ifs and fors
13:05:08sealmovethen you can script in this Nim DSL and produce standard Bash
13:05:41sealmovethe simple declarative example is just a starting point
13:06:15sealmove(of course I don't know if this is a clever idea)
13:07:15FromGitter<mratsim> the only thing you are risking is your time and also you risk to learn something.
13:08:45Zevvwise words from a wise man, this mratsim :)
13:08:59sealmove:)
13:10:09sealmovei have little opportunities to use Nim so i'll take this one
13:13:09Zevvso if I'm understanding it right, you'll be making a bash backend for nim?
13:14:17sealmoveyeah
13:15:06sealmovethough of course you won't be able to compile everything and produce legit bash, so only a subset of Nim will be valid for translation.
13:15:54sealmoveessentially it will be a Nim DSL. think of like SASS -> CSS
13:16:08Zevvlet's see how far you'll get :) I suspect you'll have fun with var arguments and such :)
13:21:32Zevvsealmove: instead of creating a backend, you also could do it at macro level, which might be a tad easier
13:22:00sealmoveZevv: that was my first consideration
13:22:45sealmoveIf it's possible I'd prefer it
13:23:27FromGitter<mratsim> try with macros first yeah
13:24:19sealmoveso macro should take AST as parameter and produce a string, right?
13:24:48sealmove(the bash code as string)
13:25:56FromGitter<mratsim> untyped means convert whatever your read as AST
13:26:15FromGitter<mratsim> but don't run Nim things on it, just give me as is.
13:30:08sealmoveBut essentially at some point the bash code must be generated. Is this done with a normal proc(smg: NimNode): string?
13:37:26*nsf joined #nim
13:38:01FromGitter<mratsim> yes you can do that
13:39:03*stefanos82 joined #nim
13:46:56FromGitter<yglukhov> Do I understand correctly that you can't use `genSym` to create a declaration of smth which can be referred after macro invocation?
13:47:19FromGitter<mratsim> {.experimental: "dynamicGenSym".} ?
13:48:20FromGitter<yglukhov> hm. what's that? can't find "dynamicGenSym" in nim sources.
13:49:00FromGitter<mratsim> ah sorry
13:49:04FromGitter<mratsim> it's bindSym
13:49:13FromGitter<mratsim> so unrelated :/
13:51:28FromGitter<yglukhov> In other words, this: https://github.com/nim-lang/Nim/issues/10706
13:56:28FromGitter<mratsim> didn't you already opened this issue 2 months ago? xD
13:58:38FromGitter<yglukhov> I don't think so :)
14:00:23FromGitter<mratsim> mmm, I think it works as intended, genSym goal is to generate unique symbols, which is useful in scopes or when shadowing names
14:00:48FromGitter<mratsim> newIdentNode does what you need
14:01:45Zevvsealmove: http://paste.ubuntu.com/p/hprmyYFwcx/ :)
14:02:46sealmoveZevv: thanks!!!
14:04:23FromGitter<yglukhov> newIdentNode doesn't cut it really because of overloading and stuff...
14:05:02FromGitter<yglukhov> it's a bit hard to define my problem in english, sorry :)
14:29:15FromGitter<mratsim> you want a unique name but that can be reused
14:39:59*kapil____ quit (Quit: Connection closed for inactivity)
14:44:08shashlick@mratsim confutils looks really cool
14:48:47shashlickAny good examples
14:55:26Zevvsealmove: http://paste.ubuntu.com/p/hprmyYFwcx/ :)
14:55:37Zevvsorry
15:00:44shashlickWhy do loaded dlls take up so much ram
15:01:09shashlick150k Nim dll takes up 12mb
15:01:17shashlickAnd I'm not allocating anything
15:01:37livcda dll created in Nim ?
15:01:45livcdor loading a dll fromm Nim ?
15:02:08shashlickBoth together
15:03:07narimiranhttps://news.ycombinator.com/item?id=19179285
15:11:07*PMunch quit (Remote host closed the connection)
15:15:11*Snircle joined #nim
15:21:40FromGitter<mratsim> @shashlick, I don't think we'll have time to document confutils for the next 3-4 weeks,
15:21:45FromGitter<mratsim> best example is this: https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/validator_keygen.nim#L29-L66
15:22:28FromGitter<mratsim> it's a json file generator, each will contain a public and private key for testing
15:24:03*chimez joined #nim
15:24:09*chimez quit (Client Quit)
15:25:29*noeontheend joined #nim
15:26:03*ng0 joined #nim
15:27:42*neceve quit (Ping timeout: 268 seconds)
15:28:30shashlick@mratsim: ya was mainly wondering about the config file locations
15:29:19shashlicklooks like registry on Windows and a serialization format on linux
15:29:27shashlickwon't be user editable
15:29:43shashlickwas hoping for a config file format for my text editor
15:32:41*dom96_w quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
15:36:21FromGitter<mratsim> confutils uses plain text files
15:37:33*Trustable joined #nim
15:38:22FromGitter<mratsim> json, yaml will be supported for sure
15:41:28FromGitter<Clyybber> Araq: What was the reason for not merging #10524 and instead clarifying in the spec?
15:43:05Araqseemed more useful to me
15:43:09shashlickokay I'll try it out once onw of those formats can be used on windows
15:51:22FromGitter<Clyybber> Araq: AFAICT it enables using negative ranges for constructing a set, right? Right now I have to do this in a for loop (correct me if I'm wrong), so I was hoping for #10524 to be merged.
15:51:41*nsf quit (Quit: WeeChat 2.3)
15:51:48FromGitter<Clyybber> Also, https://github.com/nim-lang/Nim/commit/739dc8094e4a4f6dd2f0eb0fb2eb8ce8191cb001#diff-9be874ad8ad98f21136d81b69500dd8aR13 the sentence here seems to have been broken up by that commit.
15:52:35*dom96_w joined #nim
15:53:40shashlicksize of a reference field in an object is just a pointer right?
15:55:22FromGitter<mratsim> yes
15:55:51shashlickjust tested a small dll with a small main and ram usage is minimal with loadLib
15:56:11shashlicki'm wondering if there's a copy of an object being done somewhere that's bloating my ram usage
15:56:22FromGitter<mratsim> iirc destructors changed the size of seqs, you wouldn't need to dereference to get it's length, but I may remember incorrectly.
15:56:59shashlickhttps://github.com/genotrance/feud/blob/master/src/globals.nim <= all my objects - I have a plugin object for each dll I load
15:57:09shashlickanything stands out as bloatable
15:57:37FromGitter<mratsim> TableRef
15:58:54shashlickthere should be only one instance of Ctx, which is referred to by each plugin
15:59:05shashlickare you suggesting those tablerefs will get duplicated somehow?
16:00:16FromGitter<mratsim> it just seems like the most RAM hungry object, especially the TableRef that holds Plugins references
16:00:25FromGitter<mratsim> plugins*: TableRef[string, Plugin
16:00:57*sealmove quit (Quit: WeeChat 2.3)
16:02:02shashlickstepping through the code, the loadLib() line immediately makes ram jump up, not the new(Plugin) line
16:11:25shashlickthis size issue is because of threadpool
16:11:51shashlickif i compile the dll with --threads:on, my ram usage goes up by 9mb
16:12:09shashlickan executable goes from 2.5mb to 9.5mb as well, simply for using the --threads:on
16:12:42shashlickno doubt i need threads, but it costs a lot
16:17:24FromGitter<mratsim> ugh, please open a bug report
16:18:22FromGitter<mratsim> it might be linking pthreads that just bloat the RAM or it might be another thing
16:18:39shashlickThis is windows
16:19:26shashlickI'll try without threadpool but with --threads:on and see what happens
16:25:39FromGitter<mratsim> threads on triggers things in system.nim
16:27:16FromGitter<mratsim> I'm curious about what is a distinguished threads in threadpool btw @Araq. https://nim-lang.org/docs/threadpool.html#MaxDistinguishedThread
16:29:25*Trustable quit (Remote host closed the connection)
16:29:31Araqyou pin the operations to a particular thread which turns it into a monitor
16:30:35Araqthe thread is "distinguished" which means it has an identity, you can use this for a distributed hashing key lookup table, for instance
16:30:41FromGitter<Clyybber> Araq: AFAICT it enables using negative ranges for constructing a set, right? Right now I have to do this in a for loop (correct me if I'm wrong), so I was hoping for #10524 to be merged. ⏎ Also, in https://github.com/nim-lang/Nim/commit/739dc8094e4a4f6dd2f0eb0fb2eb8ce8191cb001#diff-9be874ad8ad98f21136d81b69500dd8aR13 the sentence here seems to have been broken up by that commit.
16:30:58FromGitter<Clyybber> Sry, repasted, because I think the bridge went down.
16:31:12FromGitter<mratsim> @shashlick, I think the issue is here: https://github.com/nim-lang/Nim/blob/b6d96cafc8bcad1f3d32f2910b25cd11a93f7751/lib/pure/concurrency/threadpool.nim#L359-L364
16:32:12FromGitter<mratsim> ah no sorry
16:32:16FromGitter<mratsim> those are stack arrays
16:32:25FromGitter<mratsim> though 256 is quite large
16:32:34FromGitter<mratsim> but it should be in the BSS
16:33:02*dom96_w quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
16:34:22*dom96_w joined #nim
16:34:26shashlick@mratsim - compiling with --threads:on but without threadpool avoids the bloat
16:34:45shashlicklet me try some more tests
16:43:58*noeontheend quit (Ping timeout: 244 seconds)
16:45:20*thomasross quit (Remote host closed the connection)
16:45:35*thomasross joined #nim
16:46:26*rnrwashere joined #nim
16:50:44*thomasross quit (Ping timeout: 255 seconds)
16:50:53Araqshashlick: I think it's extensive threadlocal storage
16:51:35shashlickis that in my code or in threadpool?
16:55:43*rnrwashere quit (Remote host closed the connection)
17:00:28*rnrwashere joined #nim
17:03:12shashlicktried moving to createThread but now I just crash on startup
17:05:54*rnrwashere quit (Remote host closed the connection)
17:06:24*rnrwashere joined #nim
17:13:41*rnrwashere quit (Remote host closed the connection)
17:22:43*Trustable joined #nim
17:24:22FromGitter<Varriount> Araq: Find any good CS papers recently?
17:27:53Araqhttps://www.adacore.com/uploads/techPapers/Safe-Dynamic-Memory-Management-in-Ada-and-SPARK.pdf
17:28:22FromGitter<mratsim> Oh you missed the Ada talk about borrow checking for raw pointers at FOSDEM
17:31:43FromGitter<Varriount> @mratsim One day I'll actually be able to afford to fly to FOSDEM
17:33:27*sealmove joined #nim
17:35:45*dorelix quit (Ping timeout: 246 seconds)
17:37:28*rockcavera quit (Remote host closed the connection)
17:38:31*rockcavera joined #nim
17:40:03FromGitter<mratsim> this one: https://fosdem.org/2019/schedule/event/ada_pointers/
17:40:48*dorelix joined #nim
17:41:06*rnrwashere joined #nim
17:45:47*rnrwashere quit (Remote host closed the connection)
17:51:24*noeontheend joined #nim
17:52:16*rnrwashere joined #nim
17:53:07shashlickfeud
17:54:24*zachk joined #nim
17:56:05*natrys joined #nim
17:56:15*rnrwashere quit (Remote host closed the connection)
17:58:02*rnrwashere joined #nim
18:00:11*rnrwashere quit (Remote host closed the connection)
18:00:51*rnrwashere joined #nim
18:01:42shashlickokay have greatly reduced my mem usage but still have a leak when you unloadLib() - ram doesn't get returned to OS
18:02:39leorizeshashlick: what OS are you on?
18:02:46shashlickWindows
18:06:09leorizeso unloadLib is implemented by FreeLibrary on Windows
18:06:12leorizehttps://docs.microsoft.com/en-us/windows/desktop/api/libloaderapi/nf-libloaderapi-freelibrary
18:06:57leorize^ looking at this, could it be that the refcount never reached 0?
18:07:23shashlickwell, the dll does get unloaded - I can delete it while main binary is runnnig
18:07:58shashlickjust that it is leaving some memory around and it isn't obvious in my code where it comes from
18:19:36shashlickwell, at least I am down from 56Mb to 17Mb - cannot complain too much
18:20:29*dorelix quit (Remote host closed the connection)
18:20:50*dorelix joined #nim
18:21:18*seerix joined #nim
18:24:05*floppydh quit (Quit: WeeChat 2.3)
18:25:18shashlickwell there's a leak somewhere for sure - barebones dll without threads compiles down to 120k and leaks 1Mb per load/unload cycle
18:25:47leorizecheck if the init function of the dll allocates anything?
18:26:24*dorelix quit (Remote host closed the connection)
18:27:08shashlickthe dll is just `proc test() {.exportc, dynlib.} = echo "Hello"`
18:28:43shashlickif I don't symAddr or call test, it leaks 500k per round
18:29:59FromGitter<alehander42> i get illformed ast for obviosuly correct nim nodes
18:30:04FromGitter<alehander42> any idea what is happening
18:30:12FromGitter<alehander42> can it be a common gotcha
18:30:50shashlickokay, symAddr doesn't make any difference, calling test leaks another 400k
18:31:31narimiran@alehander42 i have reported one of those today
18:31:34shashlickfact that dynlib does nothing more than calling LoadLibrary() and FreeLibrary() under the covers suggests this is a windows issue
18:32:21narimiran@alehander42 it might be a "common gotcha", a.k.a regression
18:32:33shashlickwell, it could also be something nim put in --app:lib mode that sits in mem
18:34:41shashlickyep, something in the nim dll is causing this - i just retried the test loading/unloading pcre32.dll and memory usage is constant
18:36:43FromGitter<alehander42> @narimiran yeah because i feel this didnt happen to me before very often or at least it was more obvious why
18:36:57FromGitter<alehander42> but maybe it's just me missing something clear
18:37:29FromGitter<alehander42> but at least the error is pointing to a subnode in the ast which is a simple ident, which doesn make sense: maybe the error is just pointint to the wrong illformed part
18:37:43*seerix quit (Remote host closed the connection)
18:38:02*seerix joined #nim
18:40:12FromGitter<alehander42> i cant find your issue tho
18:42:53FromGitter<mratsim> toStrLit can maybe help?
18:43:09Araqlol, Team Nim in the Ada talk
18:44:16leorizeshashlick: dll can have autoloaded code
18:44:24*oz quit (Quit: EOF)
18:44:33shashlicki just tried with --gc:none in the dll and leak is gone
18:44:35leorizemaybe the dll initalized the GC?
18:45:10shashlickAraq: do we need the gc in nim dlls that are loaded by nim apps
18:45:52Araqshashlick: well the GC is shared and only in nimrtl.dll
18:47:59shashlickbut why do i need nimrtl when i have a main.exe which works standalone
18:49:42FromGitter<mratsim> I think a focus on dll would be nice to introduce Nim little by little in various slow languages (like Ruby, Python, JS, ...)
18:51:15leorizeany love for static libraries? :P
18:51:53*oz joined #nim
18:53:07leorizethe symbol clash of NimMain prevents two static nim library from being linked to a program
18:53:53rayman22201love for destructors lol? will solve all library memory problems guaranteed :-P
18:54:45shashlickwell, looks like I need to use nimrtl.dll but i am using threads so dead end
18:55:45rayman22201maybe gc:regions would work for you?
18:56:53*dom96_w quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
18:56:55shashlicki have a main Nim executable which loads a bunch of nim dlls
18:57:10shashlickand am using threads
18:57:11shashlickif there's a best practice, happy to adhere to it
18:59:51rayman22201idk about best practices, but it appears that you are hitting bugs when using "default gc" + loading dlls + threads. My intuition is that gc:regions may avoid these bugs.
19:00:35rayman22201either that or keep attempting to narrow down your bug and fix nimrtl.dll
19:00:38narimiran@alehander42 my issue is not in nim-repo, but in one project where i experienced that. but the author cannot reproduce it. hmmm, how old is your devel?
19:03:07shashlickrayman22201 - i'd try but there doesn't seem to be any documentation around regions
19:05:30*natrys quit (Quit: natrys)
19:06:50rayman22201@slashlick Yeah. Docs for regions sucks. This makes me sad. I through together a quick sample for @iffy recently, maybe that will help get you started: https://github.com/rayman22201/node-nim-memory-bug/blob/c6ea0d0097dcb350c020f07e97d21eeac6ae1936/clib.nim
19:07:15rayman22201through -> threw
19:09:16rayman22201Also this test in the nim core repo: https://github.com/nim-lang/Nim/blob/devel/tests/gc/thavlak.nim#L422
19:09:18shashlickinteresting, but not sure i can use it in the withRegion() style
19:09:32shashlickcause i'd load a dll and then the procs in it get called elsewhere
19:09:52shashlickam wondering how i'd use this
19:10:11rayman22201there are more advanced usages. withRegion is a template that sets the "active region" for you, but you can do it manually.
19:10:11shashlickmight work if you could init region, use where appropriate, and then free when done
19:10:20rayman22201you can do exactly that if you need to
19:11:19rayman22201https://github.com/nim-lang/Nim/blob/dee8e6e98ae868b8d933a718250c8e471bc125ea/lib/system/gc_regions.nim#L105
19:11:33shashlickbut does --gc:regions mean i have to use regions everywhere
19:11:46rayman22201it does, yes.
19:12:09*sealmove quit (Quit: WeeChat 2.3)
19:13:55shashlickokay this is a whole new world then, for a simple use case of loading nim dlls in a nim app
19:14:45rayman22201well, you are not "Just loading nim dlls in a nim app". You are "loading nim dlls in a *threaded* nim app". threads complicate everything.
19:16:24rayman22201Also, Araq generally doesn't love dlls, so there has not been a lot of testing around them beyond the very basics.
19:16:54AraqI like dlls but Unix doesn't support them
19:17:05rayman22201what is an so then lol?
19:17:27shashlicki agree but makes building real apps a lot more complicated
19:17:45Araqyou cannot put a lib*.so next to the elf binary and have it used, so you need 'sudo cp /usr/lib' bullshit
19:18:14Araqand that's so broken that I consider Unix's support for dynamic libs to be non-existent
19:18:37shashlickthere's LD_LIBRARY_PATH
19:18:49rayman22201that's politics. It's still dynamic loading code. But shashlick is using windows anway. correct?
19:19:11shashlicki want to port across eventually
19:19:30shashlickbut .so works fine for nimterop plugins
19:21:15FromGitter<mratsim> the main issue is that a nim binary loading a nim dll
19:21:25*aguspiza joined #nim
19:22:03rayman22201this is all tangent to the point. @mratsim is correct. Dynamic code loading has memory leak with the default gc it seems.
19:22:26shashlicki think its working quite well for me, except for this leak really
19:22:31Araqthreadlocal storage across DLLs is tough
19:22:32*nsf joined #nim
19:23:22Araqshashlick: try --tlsEmulation:on
19:23:40Araqand then we need to patch the stdlib to support freeing TLS
19:23:47AraqI mean to patch system.nim
19:24:04shashlickwell, my threads are localized - the app creates threads, but loads dlls in the main thread only which create their own self-contained threads
19:24:31shashlicknot sure if that makes a difference for treadlocal storage though
19:25:22shashlicktrying tlsEmulation - is that only for the main exe or the dlls too
19:25:32Araqfor all
19:25:52shashlickokay trying
19:26:14Araqleorize: the name clash can be prevented with --cppCompileToNamespace:namespace and compiling to C++
19:26:21AraqI think.
19:26:48shashlickthis is really fun though - the main editor app is just a plugin system - even the main windows are managed by plugin dlls
19:26:58shashlickso i only recompile small dlls
19:27:09shashlicksystem detects changes and recompiles changed portions
19:32:07shashlickAraq: nope - am still leaking memory
19:32:27Araqshashlick: sure but now you can free it
19:32:56shashlicki could always free the dll, but the memory allocated when it loads sits around
19:33:14shashlickby free, I mean it is no longer in use
19:34:56Araqread threads.nim line 289
19:35:29Araqyou need to do something like tlsFree(globalsSlot) except I don't know the real name for tlsFree
19:35:38Araqand it's not wrapped
19:37:51shashlickhttps://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-tlsfree
19:38:45shashlicklet me understand this - why do you think this is a thread local storage issue? how do threads come into play for dll load and unload
19:39:03shashlicki thought it was a multiple gc issue
19:39:38shashlickeither i use nimRtl across all dlls and main app or use regions
19:39:55*lritter quit (Ping timeout: 268 seconds)
19:41:30*krux02 quit (Remote host closed the connection)
19:49:13AraqI don't know what it is, nor how you do measure that you have a leak
19:51:46*rnrwashere quit (Remote host closed the connection)
19:53:49*ptdel joined #nim
19:54:39*nsf quit (Quit: WeeChat 2.3)
19:57:25shashlickjust looking at private bytes in procexp on Windows
19:57:40*disruptek quit (Quit: Bye)
19:58:02*disruptek joined #nim
20:06:35*toofly[m] quit (Quit: removing from IRC because user idle on matrix for 30+ days)
20:07:39*petersjt014[m] quit (Quit: removing from IRC because user idle on matrix for 30+ days)
20:22:29*ee7[m] quit (Quit: removing from IRC because user idle on matrix for 30+ days)
20:35:20*hohlerde left #nim ("Kicked by @appservice-irc:matrix.org : removing from IRC because user idle on matrix for 30+ days")
20:41:04*aguspiza quit (Ping timeout: 250 seconds)
20:43:58*Trustable quit (Remote host closed the connection)
20:52:02*calebwin[m] quit (Quit: removing from IRC because user idle on matrix for 30+ days)
21:00:05*Demos[m] left #nim ("Kicked by @appservice-irc:matrix.org : removing from IRC because user idle on matrix for 30+ days")
21:04:50*planetis[m] quit (Quit: removing from IRC because user idle on matrix for 30+ days)
21:13:32*noeontheend quit (Ping timeout: 255 seconds)
21:15:52FromGitter<arnetheduck> > you cannot put a lib*.so next to the elf binary and have it used, so you need 'sudo cp /usr/lib' bullshit ⏎ ⏎ you can, you simply have to enable it explicitly (through RPATH if you're the author, or LD_LIB_PATH if you're a user).. it's generally a feature that prevents some otherwise easy exploits
21:19:57Araqwell surely. Refusing to run code always prevents exploits
21:20:44*rnrwashere joined #nim
21:21:46*sealmove joined #nim
21:25:00*rnrwashere quit (Ping timeout: 246 seconds)
21:27:54*abm quit (Quit: Leaving)
21:30:29*rnrwashere joined #nim
21:31:25FromGitter<arnetheduck> how goes the ast vs ir thinking?
21:34:00AraqI stopped and all my thinking time is spent on full memory safety without the GC
21:36:31*vlad1777d quit (Remote host closed the connection)
21:37:35*vlad1777d joined #nim
21:47:50*ptdel quit (Remote host closed the connection)
21:51:15*narimiran quit (Ping timeout: 246 seconds)
21:51:34*rnrwashere quit (Remote host closed the connection)
21:56:44*rnrwashere joined #nim
21:57:38*sealmove quit (Quit: WeeChat 2.3)
21:59:35*rnrwashere quit (Remote host closed the connection)
22:23:50*Ven`` joined #nim
22:28:44*noeontheend joined #nim
22:29:41*Vladar quit (Remote host closed the connection)
22:47:23*rnrwashere joined #nim
22:48:50*rnrwashere quit (Remote host closed the connection)
22:54:16*rnrwashere joined #nim
22:55:13*rnrwashere quit (Remote host closed the connection)
22:55:24*Ven`` quit (Ping timeout: 268 seconds)
22:55:39*rnrwashere joined #nim
22:55:43*Ven`` joined #nim
22:59:33federico3want to surpass rust as well? :)
23:00:03*rnrwashere quit (Ping timeout: 245 seconds)
23:01:19*rnrwashere joined #nim
23:02:37*rnrwashere quit (Remote host closed the connection)
23:04:31*zyklon quit (Ping timeout: 246 seconds)
23:10:33*ng0 quit (Quit: Alexa, when is the end of world?)
23:14:01*rnrwashere joined #nim
23:21:09*rnrwashere quit (Remote host closed the connection)
23:21:18*rnrwashere joined #nim
23:29:49*abm joined #nim
23:31:22*rnrwashere quit (Remote host closed the connection)
23:32:45*rnrwashere joined #nim
23:44:44*Ven`` quit (Ping timeout: 255 seconds)
23:53:42*rnrwashere quit (Remote host closed the connection)
23:53:51*rnrwashere joined #nim
23:56:04*leorize quit (Ping timeout: 250 seconds)