Centralised preprocessing support

[Lokorin]

Status: Included in version 1.4.0

This suggestion idea came from a thread about making new patterns.

Problem
The parser can't parse input unless all the needed information for one pattern is on the same line or in XML. Some logs contain all the needed information but does not display it in a way that is possible to parse with DKP Log Parser (e.g. not all of the information is on the same line).

Suggestion
Some custom tailored preprocessing of logs could solve the problem. The problem with that is that it probably shouldn't be hardcoded into the program as the pattern's aren't. At the same time a syntax general enough to preprocess any contents would probably require defining a small script language, which is too much work.

Rather I would suggest letting the client interface with servers specified by the users and let the servers do the preprocessing using XML-RPC. This would let anyone who can program set up their own server and write the preprocessors that they want, as well as allowing a central server with popular preprocessors.

So basically someone would set up an page satisfying a predefined interface specification at e.g. lokorin.com/dkplp/preprocessors/ventrilo and then users could enter that location as the wanted preprocessor in the DKP Log Parser client's configuration. Then whenever the user wants to parse anything it is first sent via XML-RPC to the specified URL, preprocessed, sent back and then parsed.

[Nomad_Wanderer]

I'm okay with this as long as it remains open.

There is a pretty cool Raid tracking tool in WoW called ScenRaid. However, this tool uses code obfuscation, and forced (although free) hosting on the author's site. I don't want to host my dkp on someone else's site, or be forced into using someone elses servers..

As long as the server list remains open and server side parsing is optional, I.e. anyone can choose to run one, or just parse locally, this would be a cool addition.

[Lokorin]

Aye, I couldn't agree more with that this has to stay as open as possible. Creating a preprocessor should only be considered as a last resort when the patterns can't do all the work themselves.

The other two alternatives I considered were to
a) Have the client download preprocessors from servers once and then run them locally. That would pose a large security risk as no validation of the modules would be done, meaning that it could do anything.
b) Hardcode a set of preprocessors into the client that the user could then choose from. That would make it hard for the user to alter or add new preprocessors.

A possible c) would be to use some existing free script language, but it would have to be platform independent.

[Blipie]

[Lokorin]

I meant a script that the user could store on the local machine. E.g. there could be a folder with a number of scripts (their text would be interpreted, not compiled and run) that each represent a preprocessor on the user's own machine. Since they are scripts the user would be able to read and modify them when needed along with being able to write additional preprocessors or download new ones from a trusted source.

The problem is that it has to be possible to use the scripts platform independently (since DKP Log Parser is), it would be best if java could just interpret them directly without any additional compiled utilities. The script language also has to be limited enough so that it can't be used to for instance delete files or anything else malicious.

I don't really know what options there are out there, so I will probably have a look tomorrow.

[impishfae]

My standard raid log, assuming that I've pulled out just the raid data (and stripped out all the data before/after the raid), is usually between 10Mb and 15Mb.

That's a lot of data to send up to a remote preprocessor and then receive back. (given that most of it is combat spam: uninteresting to EQDKP)

Aside from money (given that bandwidth is relatively cheap), this will just mean that processing a log with a preprocessor will take foreeeevveeeerrrrr...

Option 1: Can we put in some sort of compression algorithm at either end? (Zipped, my 10Mb log file drops to about 1.5Mb)

Option 2: Preprocess locally (plugins! interfaces! introspection!) and then just modify the server via an rpc interface that exposes the 'common' EQDKP operations for those who want 'em, so you're only sending the relevant data over the wire, rather than the whole kit and kaboodle.

If I head off into demented dreaming land...
If we had option 2, then DKPLP could make an arrangement with someone like the Advanced Combat Tracker for EQ2 which does combat parsing. Then you could preprocess your raidlog to the ACT and generate all your battle stats as well as uploading your raid data. It's like convergent technology for MMO. Now, if only EQ2 would run on my Palm Pilot...

[Lokorin]

As usual you're correct. The problem was that I didn't find a suitable template engine, but hopefully there's something suitable in the list that you provided.

If it can also interact with EQDKP (e.g. add all members that do not currently exist with all their data entered) then that would be splendid, however I feel that writing an interface to expose EQDKP's common operations might be more work than needed.

A nice thing would be if the existing preprocessor interface could still be used, meaning that XML-RPC calls would be sent to preprocessors on the local computer. That would give plenty of options:
1. Send to a local preprocessor backed by a template engine that does some simple rearranging.
2. Send to a local preprocessor that does some kind of semantic interpretation and e.g. removes certain items.
3. Send to a local preprocessor that forks the member lines, compresses them and then sends them to a server that checks them and possibly enters them as new members.

All done via the same simple XML-RPC preprocessor interface. Now imagine adding some support for chaining preprocessors

That would mean that the log contents would first be rearranged, then black-listed items would be removed and then the member lines would be sent to the server and checked. All by chaining some simple preprocessors that could be rearranged like small building blocks. This all of a sudden feels very UNIX-like.

[Lokorin]

I have been looking at some template engines and scripting languages, but I haven't found anything that fits yet. What I'm looking for is something that can basically parse text (regexp etc is a big plus) and then produce output. It should also not give access to the filesystem, network or anything else that could be used for malisious purposes. I want to be able to know that users can download any script and then run it without having to inspect the code first.

The template engines that I have looked at so far don't provide enough power to performing non-trivial parsing. The scripting languages I have looked at seem to provide full access to everything that java can do.

*edit*

Another possibility is to run the preprocessor in a sandbox. But I would much rather find a language with just the right amount of power.

[Lokorin]

Stripping Hecl of all non-core commands and then extending it with regexp might be a possibility.

[impishfae]

Perl is the mother of all text processing languages. It's also a bit too powerful for what you're after...

It's possible SafePerl might do the job. (caveat: I've never used it myself) Safe perl is a standard module with Perl5.

Perl comes with Java bindings, so it shouldn't be too hard to embed, though at that point you'd want the preprocessor support to probably be a separate download since perl5 isn't a tiny download.

Essentially from Java you'd put in some controls around what perl files and where they lived you were prepared to use, then use the java-perl bindings to create a new 'safe' sandbox and run the provided perl code inside that sandbox.

I'll have a chat with a mate of mine who I know used Safe a fair bit (she was a Uni lecturer) and will be aware of its strengths and weaknesses. She also speaks Java, so should have the right perspective.

[Lokorin]

Yet another possibility might be to use sleep and remove the io bridge (seems like the only offending bridge from quick glance).

*edit*

But first the object manipulation through sleep would have to be turned off, I'm unsure how hard that is.

[impishfae]

This article might be of interest from JavaWorld.

I've heard some mates getting extremely keen about Groovy, though I think you're back to using Java's inbuilt policy manager to handle the security ("Groovy is integrated with the Java security model. Groovy scripts can be compiled and executed in the presence of a SecurityManager and a Policy that dictates what permissions are granted to the script.")

[Lokorin]

It seems worth a try, I guess the first step is to make a list of permissions that should be granted to the scripts. If I have understood things correctly then none of those permissions should be needed to allow the kind of code that one might use to simply process some string.

Then there's also the question about how much additional useful permissions that one might by willing to give. File read permission (java.io.FilePermission "{name}", "read") in a specific preprocessor directory might be useful, for example in a preprocessor that has some kind of blacklist. The list could be hard coded into the script instead, but that would probably make it harder for the user to edit the list (and if the list is large and used for lookups then a lookup in a separate file would be faster than reading every list entry when interpreting the script). I also noticed the following

[impishfae]

I think I got a bit lost in your post towards the end, sorry.

If I consider the specific case of the preprocessor I'd like to write: a member-list updater for EQDKP. EQDKP can store people's level and class and it'd be nice if the preprocessor could scan the log then update the server with the appropriate details.

I figured this would need a preprocessor and an eqdkp plugin for it to talk to. So dkplp passes the log off to the preprocessor who scans it, then somehow (?!) works out, or is told, what the current server is that dkplp is connected to, and it connects to the remote plugin and sends the data over.

Secondary use case and a lot more complicated: some form of combat parser. DKPLP has already worked out from the user what the current day we're talking about, is. Pass that string out, the preprocessor hands it off to a combat parser (or is a combat parser itself). Stats are generated: probably as local files in the filesystem (it would want permission to create a subdirectory in the current directory since the combat parses can end up quite prolific). User then takes those and uploads them to their webspace somewhere. Potentially even the preprocessor contacts the user and asks where they want the combat files uploaded to and it does the ftping too.

Ooo, third use case! One of our guildies asked for the loot links on eqdkp to point to ogaming, or at least provide some more information on what stats the "Uber Slashy Weapon of Fantasticness" actually had. So the preprocessor scans the file, finds all the *Loot commands, grabs the item names, then talks to something like www.eq2idb.com to find the image for that item and uploads it to the eqdkp server (which has been modified to use those images). Or it could pop up a list of loot items and present the links it thinks is the right ones from ogaming and get the user to say "Yes, that's the one" or "No, use this url here".

So I had assumed that a preprocessor would need *some* sort of remote permissions available to it, since that was the whole point of the preprocessor: it needed to munge log files, either because the log file in native form wasn't parseable by dkplp, or because there was specific functionality the user wanted to pull out of the log file that dkplp isn't set up to handle. Hmmm, some of my use cases there require popping up a GUI to talk to the user. Do you think that could present a security problem? (new window pops up that looks like some other system and asks the user for sensitive information which it runs away with)

Maybe some sort of preprocessor installer? DKPLP asks the preprocessor: I am prepared to grant you permissions to a) write to the local filesystem b) open a socket to the currently configured server c) open a socket to any server providing the user agrees, what do you need? The processor says I need a), b) and c). DKPLP asks the user: "Hi, this preprocessor you're going to install says it needs a, b and c. Tick the boxes on which ones you'll let it use. If you don't tick the boxes, some of its functionality may not work properly, here is the explanation given by the preprocessor on why it wants these permissions: [insert preprocessor security readme here]" And then the user can say what they're prepared to grant, without having to manually fiddle around with a hideous java properties file because DKPLP will set that up for them.

Then the question becomes... can you have different security permissions per preprocessor? (maybe if each one is installed in a different directory?)

. . . My head hurts.

[Lokorin]