Open Languages Need Open Test Suites

by chromatic

One of the greatest unsung benefits of the Pugs project (an experimental implementation of Perl 6) is that it has created, last I checked, some 15,000 test cases for Perl 6, with thousands more on the way.

I've realized something as I've worked on the Parrot part of Perl 6....


7 Comments

Kevin Tew
2006-08-01 16:25:49
Ruby implementers from JRuby Ruby.Net, MetaRuby, and Cardinal are trying to work together to get a ruby conformance test suite. http://groups.google.com/group/RubyTests



Kevin

Jerry Gay
2006-08-01 17:27:57
This style has been employed with great success for Parrot's Parser Grammar Engine (PGE.) PGE copied Perl 5 regex tests verbatim from bleadperl, and uses a custom harness to run them using the Perl 5 Regex interface to PGE on Parrot.


The PGE Perl 6 regex tests have been written in a similar form. This allows the Pugs team, and anybody else writing a Perl 6 regex implementation, to steal hundreds of existing regex tests. All they need to do is write their own harness to run them, as well as track which tests are classified as 'skip' or 'todo' as their implementation develops.


I find that designing tests for sharability has greatly benefitted PGE. During early development, we were able to make use of a great body of quality tests (perl5.) Now that PGE has matured, we are able to share our tests with others. I expect that as other Perl 6 Regex implementations develop, they will extend the existing body of tests, which will lead to improved quality in all the implementations.

Matthew Sporleder
2006-08-02 05:52:03
I think this transitions well into a question I asked a long time ago on the perl6 mail list (or was it a newsgroup?) about visibility into the virtual machine for administration.


Newer java's have a lot of cool stuff for monitoring memory, cpu, existing routines, etc. So.. wouldn't it be nice to monitor what your ruby/python/perl6/perlvascript was doing during runtime inside of parrot?

chromatic
2006-08-02 13:36:36
Matthew, that's a great idea. Norman Nunley and I looked into it a little bit at the post-YAPC hackathon. There is a profiling runloop in Parrot right now, but there's not much visibility from outside or inside yet.


What kind of interface would you expect to see and what kind of information would you want?

Matthew Sporleder
2006-08-03 07:27:44
Basically, the ability to run 'ps', and 'vmstat' inside of the virtual machine is what unix admins like me look for. It's what we do on (nonvirtual) machines, after all. :)


As far as an interface, that doesn't really matter. Obviously, the holy grail is to get this kind of info on-demand without performance impacts. Hooking into snmp also makes life easier for profiling with rrdtool and the like, but dumping to stderr (think java kill -3) is also pretty handy.

miguel
2006-08-06 11:34:39
Off-topic but,


All these project names are confusing. Why not steal a page from Microsoft and start calling things JavaScript.Parrot, Ruby.Parrot, Perl6.Parrot, etc. It would be less confusing and better marketing, it would be easier to explain to people what's going on when they can see the similarity to .Net.

chromatic
2006-08-07 12:53:36
miguel, I can see your point. The reason I haven't used the "official" name for any of the language ports I've worked on (Punie and Pheme) is that they're not far enough along for me to feel comfortable advertising them as compliant implementations.


If they had comprehensive test suites and some sort of specification to pass (yes, I'm aware of S5R5), I might change my mind when they reach the point of conformance.