Ana içeriğe geç

8. Content Manager#

Her tip verinin saklanması ve işlenmesi için container yapisinda mikro servisler sunar. Dosya Saklama, Metin arama, Karakter tanıma, PDF dönüştürücü, Versiyonlama, Sıkıştırma, Şifreleme, Replikasyon, Barcode servisleri barındıran dijital içerik veritabanıdır.

Neden Kullanılmalı?#

Veri tabanlarında tutulamayacak kadar büyüyen verileri merkezi ya da dağıtık yapıda saklayacak servisler sunarak yönetilmesini sağlar. Storage seviyesini yalıtarak tek bir servis katmanı limitsiz büyüklükteki farklı tiplerde medyalara erişim sağlar.

Topoloji#

CM Wan ve Lan için genel topolojik görüntü.

Genel Özellikler#

Temel özellikler.

img.png

Erişim Yetkinlikleri#

Her bir CM Repository connectorler sayesinde istenilen alt sistemlere bağlanır.

img.png

Data yaşlandırma (Tearing/Aging)#

Tearing katman özelliği ile yaşlandırma yapak mümkün. Sıklıkla kullanılan dökumanlar performans layerde tutulması sağlanır. İçerikler belirtilen kriterlere göre otomatik olarak taşınırlar.

img.png

Multi Master Topoloji#

MultiMaster yapıda içerikler karşılıklı olarak near senkron replika yapılır. Network hızına ve içerik büyüklüğüne bağlı olmakla birlikte genellikle saniye altı sürede CM Replikator Taskı senkronizasyonu tamamlar.

img.png

Dağıtık Topoloji#

CM Bölgesel olarak yerleştirilmiş sunuculara erişim sağlayarak içerik bilgisine ulaşabilir. Tüm CM ler master ya da readonly modda çalışabilir. Replikasyon yapılmak istenmeyen durumlar için geliştirilmiştir. Herhangi bir CM bağlantısı ile tüm bölgelerdeki içeriklere ulaşabilinir.

img.png

CM Servisleri#

img.png

CM Client Servisleri#

CMClient üzerinden ulaşılabilen api listesi.

public interface ICMClientService {

CMRepository addRepository(CMRepository repository) throws CMException;
Map<String,CMRepository> getRepositoryList() throws CMException;
long heartbeat(String repositoryName) throws CMException;
CMDoc addDoc(CMDoc doc, CM.CMArgument... arguments) throws CMException;
CMDoc[] addDocs(CMDoc[] docs, CM.CMArgument... arguments) throws CMException;
CMDoc getDoc(CMDoc doc, CM.CMArgument... arguments) throws CMException;
CMDoc[] getDocs(CMDoc[] docs, CM.CMArgument... arguments) throws CMException;
CMDoc[] getDocs(String repositoryName, CM.CMArgument... arguments) throws CMException;
CMDoc[] getDocs(String repositoryName, String keywords,CM.CMArgument... arguments) throws CMException;
CMDoc[] getDocs(String repositoryName, String keywords, Long docid, Integer limit, CM.CMArgument... args) throws CMException;
CMDoc[] getDocs(CMDoc doc, CM.CMArgument... arguments) throws CMException;
CMDoc delDoc(CMDoc doc, CM.CMArgument... arguments) throws CMException;
CMDoc[] delDocs(CMDoc[] doc, CM.CMArgument... arguments) throws CMException;
CMDoc parser(CMDoc doc,CM.Parser... parsers) throws CMException;
CMDoc move(CMDoc docSource, CMDoc docTarget, CM.CMArgument... arguments) throws CMException;
CMDoc[] move(CMDoc[] docSource, CMDoc[] docTarget, CM.CMArgument... arguments) throws CMException;
CMDoc copy(CMDoc docSource, CMDoc docTarget, CM.CMArgument... arguments) throws CMException;
CMDoc[] copy(CMDoc[] docSource, CMDoc[] docTarget, CM.CMArgument... arguments) throws CMException;
CMDoc mergePDF(CMDoc[] inputDocs, CMDoc dest, CM.CMArgument... arguments) throws CMException;
CMDoc barCodeReader(CMDoc doc, CM.BarCode... codes) throws CMException;
CMDoc convert(CMDoc doc, CM.CMArgument... arguments) throws CMException;
CMDoc[] converts(CMDoc[] doc, CM.CMArgument... arguments) throws CMException;
String getStreamLink(CMDoc doc, CM.CMArgument... arguments) throws CMException;
void updateDocProperty(CMDoc doc) throws CMException;
void updateRepositoryProperty(CMRepository repository) throws CMException;
void setDefaultRepositoryName(String defaultRepositoryName);
long delIndexElastic(String repositoryName) throws CMException;
long reIndexElastic(String repositoryName) throws CMException;
CMDoc[] queryFTSDocs(String repositoryName, final String searchText,final CM.CMArgument... args) throws CMException;

}

OCR Servisleri#

img.png

OCR Servisleri#

CMClient üzerinden ulaşılabilen api listesi.

public interface IOCRService {

byte[] image(String[] input, byte[] inFile) throws CMException;
byte[] convert2PDF(byte[] inFile,CM.PdfSettings pdfSettings) throws CMException;
byte[] mergePDF(List<byte[]> files) throws CMException;
byte[] ocr(String[] input, byte[] inFile) throws CMException;
byte[] pdf2SelectablePdf(String[] input, byte[] inFile) throws CMException;
byte[] tiff2PDF(byte[] inFile,CM.PdfSettings pdfSettings) throws CMException;
byte[] pdfOptimizer(byte[] inFile,CM.PdfSettings pdfSettings) throws CMException;
byte[] qpdf(byte[] inFile,String... params) throws CMException;
String barcode(byte[] inFile) throws CMException;
String typeDetect(byte[] inFile) throws CMException;
String parser(byte[] inFile,CM.Parser... arguments) throws CMException;
List<byte[]> pager(String[] input, byte[] inFile) throws CMException;
Map<String,String> imageInfo(byte[] inFile) throws CMException;
Map<String,String> pdfInfo(byte[] inFile) throws CMException;
byte[] toDoc(byte[] inFile) throws CMException;
byte[] imageConvert(String[] input, byte[] inFile) throws CMException;
byte[] clearWindozChars(byte[] inFile) throws CMException;

}

Elasticsearch Servisleri#

CMClient üzerinden ulaşılabilen api listesi.

public interface IElasticService {

ESResponse query(String index,String repositoryname,String[] jsonCriteriaList) throws CMException;
ESResponse query(String repositoryname,String[] jsonCriteriaList) throws CMException;
ESResponse query(String index,String rname,String[] jsonList, String[] inc, String[] excludeList, String[] sort) throws CMException;
ESResponse query(String repositoryname,String[] jsonCriteriaList, String[] inc, String[] exc, String[] sort) throws CMException;
ESResponse query(String index,String repositoryname,String[] jsonCriteriaList, ElasticClient.Source source) throws CMException;
ESResponse query(String repositoryname,String[] jsonCriteriaList, ElasticClient.Source source) throws CMException;
ESResponse query(String index, String repo,String[] list, String[] inc, String[] exc, String[] sort, int timeout, int start, int count, ElasticClient.Source source, boolean scrollable) throws CMException;
ESResponse query(String repositoryName,String[] jsonCriteriaList, String[] includeList, String[] excludeList, String[] sort, int timeout, int start, int count, ElasticClient.Source source) throws CMException;
ESResponse queryFTS(String repositoryName,String searchText) throws CMException;
CMDoc[] queryFTSDocs(String repositoryName,String searchText) throws CMException;
ESResponse elasticSearch(String index, String repositoryName,String field, String searchText, String[] sorter, String[] excludes, int timeout, int maxResult, ElasticClient.Source source, boolean scrollable) throws Exception;
ESResponse elasticSearch(String repositoryName,String field, String searchText, String[] sorter, String[] excludes, int timeout, int maxResult, ElasticClient.Source source) throws Exception;
ESResponse elasticSearch(String scrollID, int maxResult) throws Exception;
ESResponse add(String index, final String text,final String id) throws CMException;
ESResponse add(final String text,final String id) throws CMException;
ESResponse addCMDoc(CMDoc ftsDoc) throws CMException;
ESResponse del(String index, String id, ElasticClient.Source source) throws CMException;
ESResponse del(String id, ElasticClient.Source source) throws CMException;
ESResponse delCMDoc(CMDoc ftsDoc) throws CMException;
ESResponse del(String index, String id) throws CMException;
ESResponse del(String id) throws CMException;
ESResponse del(String index,String key,String value) throws CMException;
ESResponse get(String index, String id, ElasticClient.Source source) throws CMException;
ESResponse get(String id, ElasticClient.Source source) throws CMException;
ESResponse get(String index, String id) throws CMException;
ESResponse get(String id) throws CMException;
ESResponse getCMDoc(CMDoc ftsDoc) throws CMException;
long heartbeat(String index) throws CMException;
long heartbeat() throws CMException;
long getNextID() throws CMException;

}

CMClient#

CM Client depends initialization#

<bean id="cmClient" class="tr.com.havelsan.cm.cmcommon.CMClient" >
    <constructor-arg name="url" value="${cm.rest.url}"/>
    <property name="defaultRepositoryName" value="${cm.default.repo}"/>
</bean>

Programaticly initialization#

CMClient.getInstance(); Daha önce yapılandırılmış ise ICMClientService cmClient=new CMClient(); ICMClientService cmClient=new CMClient(url);

Auto initialization#

cm.rest.url=http://user:passwd@cmserver:port/cmapi
cm.client.ttl=300 //optional, default disable, seconds
cm.client.cacheCount=xxx //optional; cacheable max cmobjects

cm.client.cacheSize=xxx optional; cacheable max content bytes
cm.client.connect.timeout=5000 //optional default 5s
cm.client.try.count=3 //optional default 3

OCR Client initialization#

<bean id="ocrClient" class="tr.com.havelsan.cm.cmcommon.OCRClient" >
    <constructor-arg name="url" value="${ocr.rest.url}"/>
</bean>

Programaticly initialization#

OCRClient.getInstance() IOCRService ocrClient=new OCRClient();

IOCRService ocrClient=new OCRClient(url);

ocr.rest.url=http://user:passwd@ocrserver:port/ocrapi

ElasticClient initialization#

<bean id="elasticClient"class="tr.com.havelsan.elastic.ElasticClient">
<constructor-arg name="url" value="${es.rest.url}"/>
</bean>

programaticly initialization#

ESClient.getInstance() IElasticService esClient=new ESClient();

IElasticService esClient=new ESClient(url);

es.rest.url=http://user:passwd@elasticserver:port/esapi

CM ENTITY#

CM Metadata için kullanılan tablo yapıları.

CM_REPOSITORY#

private String repositoryname;
private String container;
private String connurl;
private String aclclass; //rezerved
private boolean deleted=false;
private boolean crypted=false;
private boolean readonly=false;
private boolean compressed=false;
private String cipher;
private String salt;
private String properties = "{}";
private Long reposize; //auto calculated
private Integer doccount = 0; //auto calculated

CM_DOC#

private Long docid;
private Integer version = 0;
private String repositoryname;
private Integer site=0; //rezerved
private String username;
private String filepath;
private String filename;
private String filetype;
private Date createdate;
private boolean indexable = false;
private Long docsize; //set calculated
private boolean crypt = false;
private String hash; //set calculated
private boolean deleted = false;
private String aclclass; //rezerved
private Date ttl;
private String salt;
private String cipher;
private boolean compressed;
private byte[] content;
private String properties = "{}";

CM Client Kodlama#

CMRepository repoFS = new CMRepository(); repoFS.setRepositoryname(“fsRepo”); repoFS.setContainer("fsContainer"); //db,fs,memory,es,proxy,async,sync,aging,partition,sh repoFS.setConnurl("file://tmpfs/"); repoFS.setCipher("cmCipher-B"); //optional repoFS.setSalt("1234567890abcdefg"); //optional CMRepository repoResult = cmClient.addRepository(repoFS);

CMDoc doc = new CMDoc(“fsRepo”); doc.setContent(bytes...); //byte array CMDoc v0 = cmClient.addDoc(doc);

v0.setContent(bytes2...); CMDoc v1 = cmClient.addDoc(v0, CM.CMArgument.addVersion);

CMDoc ocr = cmClient.addDoc(doc, CM.CMArgument.toPDF, CM.CMArgument.ocr);

CMDoc code = cmClient.addDoc(doc, CM.CMArgument.barCode); String barcode=code.getProperty(CM.Property.BARCODE));

CMDoc doc = cmClient.addDoc(doc, CM.CMArgument.resize1200Px, CM.CMArgument.resize600Px, CM.CMArgument.barCode, CM.CMArgument.orientationDetection,CM.CMArgument.toPDFAsync);

CMDoc doc = cmClient.getDoc(doc);

CMDoc doc = cmClient.getDoc(doc,CM.CMArgument.withContent);

CMDoc[] docs = cmClient.getDocs(doc,CM.CMArgument.withContent,CM.CMArgument.allVersion);

OCR Api Kullanım Örnekleri#

byte[] convertedPdf = ocrClient.ocr(new String[]{CM.OCRMode.selectablePDF.name()}, imgByteA); //Goruntuyu ocr den gecirip selectable pdf e donusturur.

byte[] rotatedImage = ocrClient.ocr(new String[]{CM.OCRMode.autoRotate.name()}, imgByteA); //Goruntuyu yatay veya dikey olarak cevirip uygun bir sekilde donusturur.

byte[] rotatedImage = ocrClient.ocr(new String[]{CM.OCRMode.autoRotateSelectablePDF.name()}, imgByteA); //Goruntuyu yatay veya dikey olarak selectable pdf yapar.

String info = new String(ocrClient.ocr(new String[]{CM.OCRMode.orientationDetection.name()}, imgByteA)); //Goruntunun yatayda donus acisini dondurur.

byte[] convertedImage = ocrClient.image(new String[]{"-rotate", "180"}, imgByteA); //Goruntuyu 180 derece cevirir.
byte[] convertedImage = ocrClient.image(new String[]{"-rotate", "-90"}, imgByteA); //Goruntuyu saatin ters yonunde 90 derece cevirir.
byte[] convertedImage = ocrClient.image(new String[]{"-resize", "1200"}, imgByteA); //Goruntuyu verilen pixel sayisina orantili olarak boyutlandirir.
byte[] convertedPdf = ocrClient.image(new String[]{"-topdf", "-resize 600"}, imgByteA); //Goruntuyu verilen pixel sayisina orantili olarak boyutlandirir ve pdf yapar.

byte[] pdf = ocrClient.tiff2PDF(sampleTiffImageByteArray, CM.PdfSettings.none); //Tiff image pdf donusturucu.

byte[] convertedPdf = ocrClient.convert2PDF(sampleTiffImageByteArray, CM.PdfSettings.none); //Pdf donusturucu. byte[] convertedPdf = ocrClient.convert2PDF(pdfByteA,CM.PdfSettings.screen); //Pdf donusturucu screen cozunurlugunde. byte[] convertedPdf = ocrClient.convert2PDF(pdfByteA,CM.PdfSettings.printer); //Pdf donusturucu printer cozunurlugunde. byte[] convertedPdf = ocrClient.convert2PDF(pdfByteA,CM.PdfSettings.ebook); //Pdf donusturucu ebook cozunurlugunde. byte[] convertedPdf = ocrClient.convert2PDF(pdfByteA,CM.PdfSettings.prepress); //Pdf donusturucu prepress cozunurlugunde.

List pages = ocrClient.pager(new String[]{"200"}, pdfByteArray); //COk sayfali bir pdf dosyasini 200 cozunurlugunde sayfalara ayirir.

byte[] selectablePdf = ocrClient.pdf2SelectablePdf(new String[]{"200"},pdfByteA); //Bir resim iceren pdf dokumanini ocr den gecirerek selectable pdf olarak donusturur.

String info = ocrClient.typeDetect(image); //Dosyanin header bilgisinden tipini bulur.

boolean info = Boolean.valueOf(ocrClient.parser(image, CM.Parser.checkParse)); //Dosyanin parse edilebilme bilgisini doner.

String parsedText = ocrClient.parser(anyFileByteArray);// Dosyanin icindeki text formatinda tum bilgiyi doner. FullTextSearch icin kullanilir.

Map info = ocrClient.imageInfo(anyFileByteArray); //Dosyanin metadata (exif) bilgisini doner byte[] docByteArray = ocrClient.toDoc(pdfByteArray); //Bir pdfi doc formata donusturur. (Deneysel) String barCodes = ocrClient.barcode(sampleImageWithOcrByteArray); //Goruntunun barcode bilgisini decode yapip doner.

byte[] convertedImage = ocrClient.imageConvert(new String[]{"webp", "-q 20"}, imgByteA); //webp converter byte[] deConvertImage = ocrClient.imageConvert(new String[]{"dwebp"}, sampleImageWebpByteArray); //webp deconverter

byte[] croppedPdf = ocrClient.qpdf(sampleMultipagePdfByteArray,"1-5"); //Cok sayfali bir pdf dokumanindan verilen sayfalari cikartir.
byte[] mergedPdf = ocrClient.mergePDF(samplePdfListOfByteArray); // Pdf sayfalarini birlestirir.

CMClient Api kullanım örnekleri#

CMDoc doc = cmClient.addDoc(cmDoc, CM.CMArgument.toPDF, CM.CMArgument.ocr);
0.versiyon CMDoc nesnesini kayededer.
1.versiyon PDFe cevrilerek kaydedilir.
2.versiyon ocr den parse edilen text icerigi kaydeder.
init versiyon (0) bilgileri geri doner.

CMDoc docV0 = cmClient.addDoc(cmDoc,CM.CMArgument.resize1200Px,CM.CMArgument.resize600Px,CM.CMArgument.barCode, CM.CMArgument.orientationDetection,CM.CMArgument.toPDFAsync) 0.versiyon cmDoc nesnesi goruntu 'orientationDetection' yapilarak cevrilir. Barcode init versiyonda aranir. 1.versiyon 1200 pixele resize hali ile kaydedilir.
2.versiyon 600 pixele resize hali ile kaydedilir.
3.versiyon Async olarak arka plan gorev uzerinden kaydedilir.
init versiyon (0) bilgileri geri doner.

CMDoc docV0 = cmClient1.addDoc(cmDoc,CM.CMArgument.resize100Px, selectablePDFAsync, CM.CMArgument.barCode); 0.versiyon CMDoc nesnesini kayededer. Goruntu barcode iceriyorsa properties nesnesine yazilir. 1.versiyon 100 pixele resize hali ile kaydedilir.
2.versiyon Async olarak arka plan gorev uzerinden kaydedilir.
init versiyon (0) bilgileri geri doner.

CMDoc[] docs = {cmDoc1, cmDoc2, cmDoc3, cmDoc4, cmDoc5, cmDoc6};
CMDoc target = new CMDOC("reponame");
CMDoc merged1 = cmClient.mergePDF(docs, target, CM.CMArgument.selectablePDFAsync);
pdfleri birlestirip target nesnesine store eder.

v1#