Forums

Covering OroCommerce, OroCRM, OroPlatform topics, including community updates and company announcements.

Forums Forums Oro Community Import with missing data ?

This topic contains 17 replies, has 3 voices, and was last updated by  Mike Kudelya 7 years, 2 months ago.

Starting from March 1, 2020 the forum has been switched to the read-only mode. Please head to StackOverflow for support.

  • Creator
    Topic
  • #24506

    vladimir.petrov
    Participant

    Can users import a cvs file with missing columns?
    Right now the web interface marks all records as “Records with errors”, so there is no import.

    Thanks.

Viewing 15 replies - 1 through 15 (of 17 total)
  • Author
    Replies
  • #24507

    Mike Kudelya
    Participant

    Hi

    I think no, but you can exclude column from import/export. Please read the documentation about fields configuration.

    #24508

    vladimir.petrov
    Participant

    Hi Mike,
    thank you for your answer.
    Can a pre-submit event listener be attached to a pre-existing form? In this case, the import form after users have chosen the cvs file?

    Thanks

    #24509

    Mike Kudelya
    Participant

    Hi

    Yes of course, please bind preSubmit function on buildForm:

    and implement it, $data will contain uploaded file:

    #24510

    vladimir.petrov
    Participant

    Hi Mike,
    thanks for your answer. In what file do I put ‘$builder->addEventListener(FormEvents::PRE_SUBMIT, [$this, ‘preSubmit’]);’ ? I want to re-use the existing import form — the one after which validation results show up.

    Regards

    #24511

    Mike Kudelya
    Participant

    I meant this form. Here i described how to override parts of orocrm (services, forms, etc…)

    #24512

    vladimir.petrov
    Participant

    Hi Mike,
    above you wrote “public function preSubmit(FormEvent $event)” but the documentation has it like this “public function onPreSubmit(FormEvent $event)” ? Is it preSubmit or onPreSubmit? I appreciate your answers, but so far very little real help. Also Chrome ‘find’ function does not find form extension example; you point to service.

    The oro platform can’t really import missing/wrong required columns in cvs file. In the real world, clients very often have missing/wrong data and in that situation with your oro thing users hit a roadblock and get nowhere. Do you think that is efficient?

    So, I want to generate the missing, required data, correct the wrong data and then import the full, correct csv file. That is, before the file is submitted for validation, it is corrected and after that the import works in the usual way. What is the fastest, least painful way to do it?
    Thanks.

    PS. I am not even going to go into the inefficiency of double import validation. Validating once generates errors, so those stuck with your thing wait for a long time for import. Import consistently fails for large data sets. How sad is that?

    #24513

    vladimir.petrov
    Participant

    Something along the lines of https://oroinc.com/orocrm/forums/reply/22424 ?

    #24514

    Mike Kudelya
    Participant

    Hi

    First way:

    Create listener which catch entity before validation and you can modify it (this example i have created based on Account entity):

    orocrm-application/src/Custom/Bundle/OroBundle/Resources/config/services.yml:

    orocrm-application/src/Custom/Bundle/OroBundle/EventListener/ImportAccountListener.php:

    Second way:

    But if you want to fill csv before validation i suggest you create custom bundle CustomImportExportBundle, inherit it from original ImportExportBundle, create custom form, in my case it will be orocrm-application/src/Custom/Bundle/CustomImportExportBundle/Form/Type/ImportType.php, inherit it from original ImportType.php and write in your new form something like this:

    It doesn’t matter you create preSubmit or onPreSubmit function, function name you pass in second parameter of addEventListener. Don’t forget create orocrm-application/src/Custom/Bundle/CustomImportExportBundle/Resources/config/form_types.yml with:

    #24515

    vladimir.petrov
    Participant

    Hi Mike,
    thank you for your answer — it is getting better, but not enough. Our corporate clients gave us the following incomplete and wrong data in cvs format:
    company number,Business Unit,employee ID,gender,last name, first name,street,zip code,city,date of birth,entry date,employment type
    4463,21,37,M,Steven,Jones,Biz avenue 2,33233,CityHere, 07.07.1966,1.1.1995,HTGGFG

    The clients miss email, and username; also our oro system looks for [Last name], [First name] , [Birthday], etc whereas the data has ‘last name’, ‘first name’, ‘date of birth’ and so on. How can this wrong and incomplete data be imported in the fastest way?

    Your ‘first way’ doesn’t catch last name, first name, etc because of the mismatch in column names. Can you confirm that your ‘second way’ actually works?
    Thanks.
    PS. FYI: I am getting oro bugs virtually every week.

    #24516

    Artem Liubeznyi
    Spectator

    Hi Vladimir,

    Why would you want to import wrong and incomplete data in the first place? The whole point of import validation is to not import records with missing mandatory fields.

    #24517

    vladimir.petrov
    Participant

    Hi Artem,
    as explained above, our corporate clients do have wrong and incomplete data that must be imported. They are paying for it. This is the business logic. Is that so hard to understand? Get real, guys, please.

    From technical point of view — real life data is wrong, incomplete, and dirty. Is this clear to the oro crowd?
    So what is done is actually take the good data in the database. For the “bad” data one can either fix it before import or import in special tables for further analysis. That way one gets the “good” data without spoiling it, but at the same time “bad” data can be studied/fixed. I want to fix bad data before importing it.

    The oro approach is: discard everything that is wrong and do validation twice? How slow, inefficient is that?
    I can write at least 4 math books proving how inefficient your approach is.

    The real question is: can you come with the fastest approach for solving this real life problem? So far, some partial answers, something works, something does not.

    Thanks.

    #24518

    Mike Kudelya
    Participant

    The fastest way to teach customers to send the correct data, but if that is not possible, i wrote the first way, where you can either check and modify doctrine entity before validation or the second where you can take a file through fopen and do with it whatever you want.

    #24519

    vladimir.petrov
    Participant

    Hi Mike,
    clients simply don’t have the data; that is, it is a question of data existence. So there is nothing to “teach” the clients!? If they had all the data they would have given it to us; they gave us what they actually have.

    Your ‘first way’ fails for first name and last name since our oro system looks for [First name] whereas the data has ‘first name’ as written above the other day. Do you have a solution for that in your ‘first way’ ? Is it possible to access the data file path that users opened in the submit import form in public function onProcessBefore(StrategyEvent $event) ?

    If your ‘first way’ is a partial solution, at best, why should your ‘second way’ be any better?
    When I say the ‘fastest way to solve a problem’ I mean it in the technical/coding sense.

    Just FYI: How inefficient and slow is your oro thing? I have to wait about 10 minutes for only about 6000 accounts to be imported and then there is “Import failed” error. It does not tell me why import failed, it just says it failed?!?
    Actual normal software tells users when/why/where it fails. Also oro’s import logic is: “Trow the baby out with the bathwater” ??!! This case shows it — all data is rejected because first name does not match [First name].

    Artem asked a question; got an answer and after that silence so far?

    Thanks.

    #24520

    Mike Kudelya
    Participant

    Hi

    You can map your csv headers with internal orocrm fields, write your own data converter. I think this approach will resolve ‘first name’ problem.
    We know about import performance, we will start working as soon as possible.

    Thanks for your patience.

    #24521

    vladimir.petrov
    Participant

    Hi Mike,
    can you give an example of how one registers a custom data converter as a service in importexport.yml?

    Your link lists code like this: return [
    ‘item_id’ => ‘originId’, ….

    item_id refers to ‘first name’ or [First name] in my case?

    Oro code is poorly documented internally as well. There are things like
    /**
    * {@inheritdoc}
    */ but no comments there. If there were comments some of the questions will be clearer. Let’s not mention that up-to-date, internal documentation is a hallmark of good, robust code. Also your life will be easier if there were internal documentation.

    Also by having to write my own custom data converter I lose the benefits of code re-use — another slow-down.

    Yeah, they always say that they will fix it soon and it never happens.

    Import is a basic operation and when it is messed up people who use oro suffer the consequences.

    Thanks

Viewing 15 replies - 1 through 15 (of 17 total)

The forum ‘Oro Community’ is closed to new topics and replies.

Back to top