Posted: Tue Aug 16, 2005 9:57 pm
Well, it's been a month and a half since version 1.3.1, so you must have guessed that a new version was in the works.
A couple of the things you can expect are integrated PDF support and a SOAP source object.
There are a few details I'll save as surprises, but now that it's spending more time in testing than in development, I figure it's worth letting folks know that a new version upgrade is in the works, and I hope you enjoy the new features!
As always, I welcome your feedback on how Anthracite can be made better to meet your web mining and text processing needs.
Thank you!
Joe
Joe Pezzillo, Founder
Metafy LLC
Boulder, Colorado USA
Posted: Thu Jun 30, 2005 4:48 am
If you're a fan of the long tail model, here's Anthracite's version.
The Y axis is "development time" and the higher you go, the more it costs. The X axis is "compute time" and the farther right you go, the less expensive it gets.
Traditionally, development of custom text processing systems has been the domain of expensive specialists, for example, Perl programmers. I was a Perl programmer once, sure, I was young and I needed the money, but it was great work and at top shelf rates. These types of jobs are something of a hold over from the days when compute time was expensive, everything was a black box and you needed a bona fide wizard to get the most shebang for the buck.
Now, it's development time that's expensive (hence offshoring), and compute time continues to give Moore's law a run for the money.
So you end up with code or script based solutions that require custom development in the expensive upper left zone of the long tail chart, and Anthracite in the lower right zone.
What this means is that where previously the application had to be of sufficiently high value to warrant the development expense, now many more areas (the long tail) can benefit from customized web data solutions, as well. It also means more iterations can be done, new ideas tried with much lower risk, and all the other benefits that you'd expect from bringing the access to the tools to a broader audience.
Now that the base toolset is almost complete, the next step is to make sure that Anthracite is a more useful solution to growing audiences in the early adopter markets like financial analysis, SEO and even biotech (funny, I always thought it would be web developers who would benefit the most from Anthracite, I'm thrilled to have learned about these other areas). In particular, one thing I'm looking forward to are new ways to let Anthracite users offer their solutions to each other and to customers.
I always welcome your feedback on ways to improve Anthracite to meet your existing and coming web mining needs, and hope you will let me know about more of the fascinating work that you are doing with it!
Thanks for your interest!
Joe
Joe Pezzillo, Founder
Metafy LLC
Boulder, Colorado USA
Posted: Thu Jun 30, 2005 4:19 am
Okay, some of you know I'm sensitive. So when I read Theolo's post on VersionTracker saying that Anthracite "freaks out" on files larger than 30MB, I noticed. Of course, then, when he called, I got the full story.
Basically, he had a 65MB table. Yes, a large 65MB HTML table, well, actually, it was more like 3,500 tables, all in one document, with one large outer enclosing table but mostly a lot of little tables within that. And, sure enough, when Anthracite started unwinding the table tags, it choked after about 30MB. Turns out, the problem was that one of those Cocoa autorelease pools needed to be setup inside one of the loops to help drain the memory being created by the autoreleased string objects being returned from foundation. It's all in the Cocoa documentation, as it turns out, too. Ahem. And Amen! It seems to be working as desired now.
Here are the results on the 1.4Ghz mini, while I was doing other stuff in the background, too:
65MB source HTML file parsed into 3572 tables, then dropping the last one (with the new excerpt indexing in 1.3.1), done in 632 seconds, or about 5 1/2 tables extracted per second. Sure, it would be faster in C and maybe even Perl, but it only took a couple minutes to design the extraction system and almost anyone can do it. (See Anthracite and the long tail model)
And, the crux of the story is this: Anthracite works much better with large input files now!
Screen shot to follow.
Posted: Fri Jul 01, 2005 11:27 pm
As a special treat for those of you who actually read the forums, here's the first public announcement:
Version 1.3.1 of Anthracite is now available for immediate download.
Version 1.3.1 features:
*Improved Excerpt Processor - now with negative indexing support
*Improved Large File Support - tested to parse thousands of tables from more than 50MB of HTML in a single file
*Improved MySQL Multiple Export - fixes problems with source insertion, trim, repeat and ignore array padding handling
*Improved Regex Handling - fixed corner case where match on last character sometimes caused loop
I hope that you find version 1.3.1 to be the best version of Anthracite yet, and will let me know how Anthracite can be made better for your web mining needs.
Joe
Joe Pezzillo, Founder
Metafy LLC
Boulder, Colorado USA
Posted: Fri Jun 03, 2005 2:39 pm
Get it while it's fresh! Version 1.3 is now shipping, and features the following new functionality and enhancements:
MySQL Batch Queries - now you can set the number of queries to be performed at a time using the "Batch Size" setting on the MySQL Export Object's "Host" configuration tab
Split Join Setting - Instead of being based on the type of input, now you can explicitly specify which function you want the Split/Join Processor to perform. ***THIS MAY REQUIRE YOU TO UPDATE YOUR OLD PROCESSES*** The default setting is set to "Split On" so if you're using the Split/Join processor to Join an array of elements into a string, you will need to set that!!! This was discovered in the sample documents, and we hope we've fixed them all.
XML BR Tag in UnifyLF - you now have the option of unifying linefeeds to the XML-valid "<br />" tag.
Built-in HTTPS - CFNetwork's built-in https support in now available in both the URL List and the URL List File Source Objects.
Improved Encoding Support - fixed a corner-case bug that prevented some HTTP header encoding information from being used.
MySQL Multiple Source URL Inserts - fixed a bug that prevented multiple source URLs from being inserted properly
Improved Strip Tags - improved the performance of the Strip Tags processor (had been saving the stripped tags, now skipping that step).
And of course a few more much smaller details under the hood.
NOTE: This may be the last "Panther-compatible" version of Anthracite so that we can focus more attention on the new capabilities in MacOS X 10.4 Tiger.
Please don't hesitate to contact me directly if I can help you get set up and running with Anthracite, or otherwise answer any questions.
I've been thrilled to learn about all the amazing different ways that customers are using Anthracite, I hope you will please let me know if there is any way I can make it better to meet your needs.
Hope to meet some of you at WWDC next week, too!
Joe
Joe Pezzillo, Founder
Metafy LLC
Posted: Sun May 29, 2005 9:39 pm
So now that everyone is off of Jaguar finally, how do folks feel about going Tiger-only? I'm thinking we'll get all the current updates and a couple new things into a 1.3 release, and then make 1.4 our first Tiger-only release so the numbers line up a little.
Built-in PDF support is waiting in the wings!
Let me know, and we can also keep the Panther-compatible version around, but of course I'm eager to start integrating more Tiger-only functionality, too.
I won't bother with a goofy poll this time, if you are concerned about Panther support, please feel free to contact me directly.
Joe
Posted: Tue May 17, 2005 8:31 pm
Today (5/17/2005) is the one-year anniversary of the public launch of Anthracite Web Mining Desktop Toolkit software, and what a fantastic year it's been thanks to all the users whose support has helped us keep pushing it forward and whose real-world applications guide us in making it better.
Speaking of better, check out some of what's in the new version 1.2:
AppleScript!!! - A bunch of new Source Object settings have been made accessible via AppleScript: CURL, Links on Page, Static Text and the URL Generator can all now be configured via AppleScript! Plus, we've also updated the accessibility of your Results Objects in AppleScript, so that you can get all the results in an object as a giant string, or as an array of individual strings representing your results. Oh yeah, and you can also toggle the sound when done. Examples of all of this are included with the new distribution disk image. Yes, much of it is anticipation of the Automator Actions for Anthracite that we hope to ship later today, too.
But that's not all, we also enhanced the Spotlight support so that your search term should be highlighted or shown in context in your document when it's opened from a Spotlight search! Playing a sound when done is now a preference and the Results Window got a little fix up under the hood. As did the easter egg, but then, you probably hadn't noticed that it needed it, had you?
Anyway, some major updates hence the minor revision number change.
As always, please let me know if there is any way we can make the software better for you and your needs.
I truly appreciate your support and hope that you find this to be the best version of Anthracite yet.
Sincerely,
Joe Pezzillo, Founder
Metafy LLC
Posted: Tue May 17, 2005 8:49 pm
People seem to always doubt any easter egg is in there, but here's more evidence, a "portrait" of yours truly:
http://www.metafy.com/anthracite/testing/egg/yours_truly.jpg
(yes, we QA test the easter egg, too!)
Posted: Mon May 16, 2005 8:20 pm
Well, we've got the final candidate for 1.2 baking right now. There are quite a few new features instore, especially for AppleScript (and, ahem, wink wink, Automator) users.
However, we're finally hitting the point at which if we want to include some Tiger-specific stuff, we're going to have to drop support for 10.2.8.
The existing version (1.1.2, build id M126) is Jaguar compatible, and we'll be glad to make sure it remains available for awhile if you need it. But, since we can no longer build the functionality on Jaguar (e.g., Spotlight query support within an Anthracite document), we're just going to have to move on...
PLEASE LET ME KNOW if you are going to miss 10.2.8 support, and if not, let's all raise a toast to the classy cat that was Jaguar.
Joe
Posted: Fri May 13, 2005 4:52 pm
Howdy!
We're almost certainly going to have to drop support for MacOS X Version 10.2.8 ("Jaguar") as we being adding Tiger-specific functionality.
Just in case, we wanted to get a feel for which versions of the OS you're running, to see if there are still many folks depending on 10.2.8 (we hope not, but you never know)
Thanks!
Joe
Posted: Mon May 09, 2005 2:48 pm Version 1.1.2 - Bug fix for Google API Source Object
Based on internal testing and initial reports from pre-release test users, I believe that the Google API Source Object bug that caused the crash has been fixed.
Of course, it's only fixed if it works for you, so I hope you will please let me know if it is.
Also, once again I think that the performance of the internal task handler (used by UNIX Commands and other processes) has been improved, in particular, a couple of potential blocking io issues with certain UNIX commands should be resolved, and has been tested on 10.3.X and 10.4.
I hope you will share your results.
Finally, check for updates in the Sample Documents, including the addition of a "Proper Nouns" sample document that includes a ranked list of proper nouns extracted from the King James Bible.
Please don't hesitate to contact me with any questions or feedback, I truly appreciate your interest in Anthracite and hope you will let me know how I can make it better for you.
Joe Pezzillo, Founder
Metafy LLC
joe(a)metafy.com
Access Information, Distribute Knowledge (tm): http://www.metafy.com 1.1.2 Follow-up: KJV Bible Proper Nouns Posted: Tue May 10, 2005 9:07 pm
Sorry, looks like the version of the Proper Noun sample document that got included with the final distribution disk doesn't have the King James Bible ranked capitalized word output afterall, so here it is viewable on the web:
http://www.metafy.com/anthracite/samples/output/KJV_RankedCapitalizedWords.txt
It's just a plain text file, but it's the most basic raw output available directly from Anthracite, here are the first few results:
Code:
and here's a a screenshot of the Anthracite process that created it:
(click for larger)
Posted: Mon Mar 28, 2005 5:12 pm
Announcing Anthracite 1.1!
Released 3/28/2005
Anthracite 1.1, Build ID M124, features several important new features and a number of performance and stability enhancements, including several fixes for user reported bugs.
NEW: Google API Overhaul, now featuring AppleScriptable Google searches Newly included sample documents and AppleScripts demonstrate a wide variety of Google tasks, including automating keyword ranking reports. The Google API Source Object now makes use of the more contemporary CoreServices WebServices methods to perform SOAP queries and no longer uses the older AESend infrastructure. This means one important change to existing Anthracite processes, now "raw results" are returned in Apple's PList format instead of XML. A separate document explains the difference, what it means for your processes, and how to make the needed changes.
NEW: Cool UI Trick "Drag and Drop Connections" between existing objects. Now, instead of having to manually disconnect and reconnect a process chain to insert a new processor or report, you can simply drag the new processor or report onto the line representing the connection. Makes editing your processes EASY! Check out the video clips in the previous posting on this.
Updates & Bug Fixes: Performance improvements in several text processor core functions including text between, in some cases approaching 90% performance improvement; fix for Google API UI ComboBox data source nil/empty value handling; attempted fix for user-reported online/offline condition that may cause damage to saved preferences; new sample documents included in the distribution disk, including sample AppleScripts for use with the Google API Source Object, and additional online documentation including sample video clips of the new UI trick.
Anthracite 1.1 is available for immediate download from the standard distribution link provided by e-mail, if you have not received or no longer have your download e-mail, please sign up for it now at:
http://www.metafy.com/downloads/anth_download.php
Anthracite 1.1 requires MacOS X 10.2.8 or newer.
New users are invited to try Anthracite for a FREE, fully-functional two-week trial of the software. Licenses for Anthracite can be purchased online for US$99 per seat. Educational discounts and site licenses are available.
Posted: Sat Mar 12, 2005 3:19 am
Well, as with so many ambitious blogs, it's been kinda quiet around here, eh?
But just so you know it's not quiet at the compiler, here's what's coming in Anthracite 1.1:
1) New: "Drag Connect" objects to add processors to existing process chains.
See this in action in these QuickTime movie screen shots:
"Drag Connect" Sample Movie - Adding Processor to Existing Chain
"Drag Connect" Sample Movie - Modifying an Existing Process Chain
Pretty cool, and thanks to Ambrosia Snapz Pro, you get to see it here first.
2) The second thing we've done is overhauled the Google API engine . More than a few of you noticed that the old-school AppleEvent SOAP methods were, ahem, unstable. Thanks to the recipes in Joe Zobkiw's book, we've converted this to use the CoreServices.framework, now it seems to be MUCH, MUCH more stable. HOWEVER, it completely changed the format of the Raw XML output, now the "Raw Result Data" is in more of a plist-style format. A big change, but the improvement in performance and reliability will hopefully make any required modifications to your existing Anthracite documents a one-time minor annoyance. I'd like to know how to get the raw XML out of the CoreServices result, but I haven't found a way yet. Might have to wait until WWDC.
We were going to call it 1.0.10, but that seemed hokey, with the new object drag connections and the major overhaul to the SOAP foundation of the Google tool, it's a big step forward. More features are still on the drawing board and can be expected soon. This is the 10th update in less than a year on Anthracite, and the product continues to serve more and more customer's needs. If it isn't meeting yours, you can be sure I want to know about it, so please drop me a line anytime!
Thanks for your interest in and attention to Anthracite and Metafy!
Sincerely,
Joe Pezzillo, Founder
Metafy LLC
Posted: Fri Dec 17, 2004 1:02 am
Yes, it took longer than expected, but version 1.0.9 is currently available. If you are getting a "version expired" message with 1.0.8 or older, you'll want to update to this version ASAP to continue using the software. To update, simply download the new version, and copy the Anthracite icon into your Applications folder, replacing the previous version of Anthracite.
Updates in Version 1.0.9:
AppleScriptable Source URL List
Improved Handling of Mixed Arrays in Excerpt, RSS output, Apache Template processing, and other areas.
Simple Command Line Help (anthracite --help)
Product URL not UserAgent in XML output for validity
Clarification and Cleanup of various Error Messages
Posted: Wed Aug 25, 2004 6:33 pm
I guess the RSS bug has bitten pretty hard over here at Metafy, today we've got two announcements, both with the RSS buzzword fully enabled.
Anthracite Web Mining Desktop is now at version 1.0.8 and enhances the RSS capabilities over the last version with better handling of single text results and a fix to the bug that caused certain multi-source processes to always output RDF.
Additional user-reported and testing discovered bugs have also been updated, including the "double offset" line drawing bug, better handling of relative links in the Links on Page Source Object, error messages when the Google API Key is empty, required completion of URL and UNIX cmd table entry prior to closing/saving an edit sheet, and other minor problems and touch-ups that were reported or found during testing.
Also, please give Stickler! a try. It's a skinnable RSS Feed transparent headline ticker for your desktop. Just drop one of those "RSS" badges on it and voila, you're watching the feed.
There's a nifty example that shows how to use Anthracite to generate an RSS Feed of System Status information that's then viewed in a Stickler! window, with both the Anthracite output and the feed source able to be updated regularly.
http://www.metafy.com/anthracite/docs/examples/rss_feeds/
As usual, your feedback is incredibly valuable, so I hope you won't hesitate to share it with me.
Joe
Posted: Wed Aug 11, 2004 7:58 pm
Kinda quiet on here lately eh? Well, here's why:
Version 1.0.7 is almost finished, and it's got one major update that I think you're going to like, and many smaller touch-ups and fixes, some that you might have been waiting for.
Top Change: Enhanced RSS Generation
There were a couple of disconnects internally with the Export Object that are now fixed, and the difference is dramatic. Now, it's entirely possible that Anthracite is the easiest way to make an RSS Feed from any web page. There's a page of examples that show you what I'm talking about:
Easy RSS Generation with Anthracite
Other important additions, changes, updates:
HTML Entities conversion processor subtype
Auto-scroll Results Object to newest, expanded results.
Google API Source Object "Raw XML" output
Fix to Basic Report Object ordering bug
Command line "-d" Debug option
There are some smaller details under the hood, but hopefully these improvements help.
There is also going to be a new "blog-like" column on the Metafy homepage, and that will help keep the tempo up!
As always, please let me know if you have any questions or feedback! I truly appreciate hearing from users and hope the software is useful for you.
Joe
Posted: Fri Jul 09, 2004 12:58 am
1.0.6 is posted to the various places where you can normally get it.
A couple notable changes:
* AppleScript "run" support so you can now script your Anthracite documents processing (such as with an iCal event).
* Macintosh Consultants can now join Metafy's Affiliate Sales Program and earn 25% of the sale of the software to their clients and customers, see http://www.metafy.com/affiliates/ for more information.
* Additional changes include better handling of encodings in large documents, an added feedback menu, Unicode/UTF-8/ASCII handling for UNIX Commands, and added URL Generator "Small Business Database" Example to online documentation.
Posted: Fri Jun 18, 2004 7:08 pm
We take our user's bug reports seriously, and therefore we're always glad to announce that we've fixed them.
Look for version 1.0.5 coming up early next week, fixing reported issues with HTML Tables, Strip/Trim, and enhanced support for server-reported character set encodings.
Thanks as always for any reports of bugs or problems, and for all your feedback!
Joe
Posted: Mon Jun 14, 2004 6:09 pm
Version 1.0.4 (Build M118) is now available for download.
The key update is that the Unicode support has been updated throughout the application enabling web mining in many languages. The distribution disk image includes samples in Chinese and German, and the software has been tested against Japanese, Thai, Arabic, Hebrew, and several European languages.
Also, additional status messages have been added to the UNIX Command Processor Object.
Documentation has been updated and enhanced on the website.
Please let me know if you have any questions or feedback! I'm eager to see what folks are able to do with this.
Joe
Posted: Wed Jun 02, 2004 2:43 pm
1.0.2 is available for download.
There are more than a dozen changes, including:
Additional Control Character Handling in Wrap and Custom Snippets
Enhancements to Combined Output in File and MySQL exports
Fix to "Basic Report no Inputs in Table View" bug
Live Update of Results Objects
Work on Google Key handling in Keychain
Current Version indication in Splash Window
Log Window improvements
As always, please let me know if you have any questions or problems!
Joe
Posted: Tue May 18, 2004 4:51 pm
Amazing how an external deadline can force you to get something done, and what a great whirlwind it can be.
Anthracite 1.0 is released in time to enter the Apple Design Awards.
There were a bevy of changes during the last week, and an interim release had been planned to incorporate the many bugs that testers commented on, and feedback from an Apple HI review.
The latest version will be available via the homepage download/e-mail form and at https:// secure.metafy.com /dist/anthracite/
Please give it a whirl and let me know what you think of the changes!