With multi-core CPUs being the standard, it would be nice if it were possible to indicate that a project is built with multiple threads. Now 7 of my 8 (Intel i7) cores are sitting around doing nothing while I wait for one thread to compile everything.
that would be multi-process since the main work is done by the avm runtime, proven that multiple run-times can be started (database connections limit maybe) then this should be possible... //TODO: check that out
You are asking for the wrong thing. Be careful what you wish for.
You are assuming that multi-threading will solve your problem. Perhaps it will and perhaps it won't. Better if you state the problem you wish to be solved (compiling takes too long) as there may be better ways solving it. Also, it would be helpful to know what compiler speed would be acceptable to you (needs to be twice as fast, 10x faster, etc).
I don't mean to pick on you but I just finished reading a list of 20 or so enhancements that a customer claims they want. Only one or two of them make any sense at all. The rest are solutions to unstated problems.
My 'project' (approx 8000 sources totalling 325 mb) takes two to three hours to compile resulting in approx 900 mb r-code for one Progress version (of two) for one database platform (of three) (luckily with OE11.0 no more 32 / 64 bit compiles). One core is being hammered while compiling (which is fine by me, that's what it's there for). Seven cores are doing little to nothing. So yes, it would be nice if the compile process were quicker - the easy to make assumption is that actually recognizing and using available resources could help in a quick win way. When more than one core would start compiling then other bottle necks than cpu may surface - disk, memory, network, whatever - but with only one compile process running we'll never know.
Note that the actual deployment build process is spread over two (virtual) machines - one for 10.1C and 10.2B the other for 10.2Bx64 - and could (if it were worth the effort and compile time were more critical) be distributed more. Since the build runs in batch mode on servers it's not something I'm staring at and waiting to complete so that I can get on with being productive.
Of the 325 mb code mentioned above, 300 mb is generated code - when generating this code the number of generating processes to start can be indicated. Combining the results of multiple progress sessions requires a bit of administratie paper work, but nothing difficult.
(title also changed from multi-threaded to multi-process - thanks to Marian for pointing that out).
gus wrote:
You are asking for the wrong thing. Be careful what you wish for.
You are assuming that multi-threading will solve your problem.
Gus, the thing Stefan is asking for is a multi-threaded builder in Eclipse (OEA)... adding multi-threading in Progress runtime is something different and partly I have to agree with what you're saying. Be careful what you wish for, multi-threading is a powerful beast but needs extra care when used, probably not the most important thing for a business application anyway.
As a hint, the JDT team got the same request at some point and I can say this is working quite well now... I usually see half of my cores used when building a project clean, the number is configurable but for me giving away four core to have the entire workspace built within minutes makes sense.
I am out of the office until Monday April 9th. I will respond to e-mails when I return.
heh. I wish for out of office messages to be banned
Hi,
I guess that you're not compiling the full project every day on your workstation, so I assume this is on your build servers. In this case, you can try using PCT to automate your builds, and especially PCTCompileExt with numThreads attribute. This will start multiple OE sessions to compile your code. This may reduce the duration of a full build...
Gilles
I agree, that I'd like to see more of those 8 rather expensive cores busy...
But I wonder how easy that will be with OO systems where you may need to compile a bunch of other classes first before you can compile a single .p or .cls
So I see a risk of multiplicating the actual work of compiling base classes of interfaces when you don't follow the dependency path. How to coordinate that with distributed processes?
Laura Sterns and Peter Judges presentation last week on the OO internals was very interesting on this regard.
Are you using GUI for .NET? Avoid having network folders on you propath at all. They kill performance because by default all .NET types are seeked in the propath first. With network drives that kills performance.
Then use local (may be empty) DBs for compilation and use a good -B value
Eliminating the network can have a 10 times impact on compilation performance.
Of course that's something PSC might optimize as well...
Gilles,
we have a "new" full object orientated application. (.NET)
Sometimes, with OEA, we have still compile errors after a full compile.
One solution, mostly after changing TT definitions files (include), we have to run a full compile twice or more to eleminate "errors".
Another solutions is to "clear" all OE errors and run a full compile again.
If one single compiler already have problems, what will be happen when this is done by mutliple compiles?
Stefan
No, I'm not building my 'project' every day, but I do need to initially build it when getting started and will probably need a quite a few more complete builds to catch up since we make heavy use of includes. My current case is that we have a new version with database adjustments which is configured as a new project.
I have glanced at your builder tools a few times - but the technology is too far out from my progress skillset - and my java skillset is pretty much non-existent - so I've only read your documentation. Our build process is completely in progress - this I understand - and it works well enough.
I always remove all R-Code before I do a full compile using a simple batch referenced as an external tool.
That solves the issue you are describing.
why not using the "clean" instead, that should remove the r-code isn't it?
the real benefit of pct is that it can poll your version control repository
, check for updates and build automatically, with all errors emailed to you
on an exception basis. So you are always aware of any potential error
without having to manually initiate the build process
Julian
marquardt wrote:
If one single compiler already have problems, what will be happen when this is done by mutliple compiles?
Stefan
I would think that the maximum number of builder processes would be an openedge project property.
jmls wrote:
the real benefit of pct is that it can poll your version control repository
, check for updates and build automatically, with all errors emailed to you
on an exception basis. So you are always aware of any potential error
without having to manually initiate the build process
Julian
Oooo, I may have to sharpen my java skillset then...
We actually have MS TFS 2010 implemented by corporate - and this is actually already building for us when changesets are checked in. This 'build' is currently only replicating the changeset to our file share which is used by our nightly scheduled builders.
jmls wrote:
heh. I wish for out of office messages to be banned
I so wanted to mark this answer as the correct answer... ;-)
Julian, true for a tester.
But a developer needs to know if his changes are ok before checking in. Otherwise you have even more dirty code in you repository.
Mike,
thanks for the info.
This should be done by OEA by default, isn't it?
I hate professional products that needs external tool to do the job we all expect.
Perhaps, now, when Progress will become "real progres" again, OEA would become better.
But MS is miles away with the new Studio.
Integrated version tools, sprint planning, team planning, testing tools, and so on, all integrated.
BTW:
You can do write very fast a small Eclipse Plugin to do this job, and integrate this as dropin.
Stefan
not if you use a dvcs structure so that you have a "personal repo" , a
"developer repository" , a "qa repository" and a "release repository"
so when work is pushed from your repo to "developer", a build gets done and
you get notified.
if all is well and good, the updates are then pushed from developer into qa
if all is well and good, the updates are then pushed from qa into release
we use the developer repository for code review , so the workflow is
personal->devel (code review fail) -> personal
personal->devel (code review pass) -> qa (qa fail) -> personal
personal->devel (code review pass) -> qa (qa pass) -> release
YMMV
Julian
This should be done by OEA by default, isn't it?
Yepp.
Perhaps, now, when Progress will become "real progres" again, OEA would become better.
We'll see.
But MS is miles away with the new Studio.
Behind or ahead
Integrated version tools, sprint planning, team planning, testing tools, and so on, all integrated.
You can have that with OEA as well. We use that.
mikefe schrieb:
Integrated version tools, sprint planning, team planning, testing tools, and so on, all integrated.
You can have that with OEA as well. We use that.
What is the name of the Progress product?
What is the name of the Progress product?
I never wrote of a Progress product. OEA is built on top of an open eco system that provides a lot of (and very often free) functionality. Did you check out MyLyn and its integrations for task management and planning?
But if you're interested in that kind of discussion, you should start a new thread. This one is already hopelessly hijacked.
One solution, mostly after changing TT definitions files (include), we
have to run a full compile twice or more to eleminate "errors".
Another solutions is to "clear" all OE errors and run a full compile
again.
Certainly for OOABL cases, I've found that setting the build order of projects helps a lot. It's under Preferences > General > Workspace > Build Order.
-- peter
Certainly for OOABL cases, I've found that setting the build order of projects helps a lot. It's under Preferences > General > Workspace > Build Order.
If you have the habit of having tons of projects... But one might expect that the IDE should be able to compile a single large project without and tricks and traps as well.
Stephan,
we have a "new" full object orientated application. (.NET)
Sometimes, with OEA, we have still compile errors after a full compile.
One solution, mostly after changing TT definitions files (include), we have to run a full compile twice or more to eleminate "errors".
Another solutions is to "clear" all OE errors and run a full compile again.
PCT (and PCTCompile, the single-threaded compiler) handles incremental builds, and so handles db schema upgrades, tt definitions changes, and only recompiles what's needed.
PCTCompileExt may have some issues regarding multi-threading, YMMV. PCT is open-source, don't hesitate to report bugs (or provide patches) ! Test cases more than welcome...
Oooo, I may have to sharpen my java skillset then...
No need to have a deep knowledge of Java, you're just using it when you execute Ant. This can be reduced to these steps :
Ant is written in Java (and so is PCT), and there are a good number of tasks to deal with Java problems, but apart from that, it's XML syntax wih no relation/dependency on Java.
But my point of view may be a bit biased, as I'm working with Java since many years... If you have problems setting up, don't hesitate to open a bug on the Google code page.
Gilles
Marian,
Sure, I know what he was asking for. I just want the /problem/ to be clearly
stated. Suppose we multi-threaded the compiler in some way and that resulted
in a speedup of 5 percent along with some other benefit. That wouldn't help
Stefan at all.
>> heh. I wish for out of office messages to be banned
Sorry about that, I didn't intend for the auto-responder to answer anyone outside my employer. I changed my settings so it won't happen again.
wow. A responsible citizen ! welcome
& thanks !